Spring boot Webservice / Microservices and scheduling

I am having a spring boot application which exposes REST APIs and also am planning to add a spring boot scheduled task.

The purpose of task is to pick up some pending 'reservations' from data store which is momgodb and process it.

Now the question is when multiple instances of service running, there is high probability all the nodes will pick up same 'reservation records'.

I was looking at mongodb lock for this and was checking updateAndModidy / transactions (in 4.x) of mongodb.

Also possibility of separate lock table with only one record, where in I lock the record and perform operation and unlock record. But this way at a time only one node will be working and also overhead of cleaning up lock if unlock operation failed.

So question is regarding multiple instances of services running and scheduling the tasks as mentioned above. I would appreciate and grateful, if anyone could point to best practices or solution around this topic regardless of Technologies (spring boot / mongodb etc).

Depending on your deployment, a solution that works for us is to create a cluster with the nodes from the same application using Hazelcast and then when the task is run, all the nodes check if they are master and only the master runs the task.

You can read an intro about Hazelcast in Spring here: https://josdem.io/techtalk/spring/spring_boot_hazelcast_es/

What we use is Hazelcast with Consul so we don't have to configure the ips/ports manualy, using this: https://github.com/bitsofinfo/hazelcast-consul-discovery-spi

If you don't have many (dynamic) nodes, it's a relatively simple solution that has been working fine for us (3 nodes, coordinated zero-downtime deployments with consul/fabio from Jenkins)

In the past we used to let Quartz scheduling coordinate all nodes, using a common JDBC datasource for Quartz as synchronization mechanism, but the implementation is faulty and every now and then we would end up with TABLE locks that we would have to unlock manually, killing some nodes.

Cheers

Edit:

This is how one of our class looks:

// Optionally autowired, in case we don't run in a cluster (development)
private final Optional<HazelcastInstance> hazelcast;

	@Scheduled(cron = "${refresh.cron}")
	public void scheduledRefresh() {
        // Run if we are not in a cluster or 
        // we are the first member of the cluster
		if (!hazelcast.isPresent() || 
   hazelcast.get().getCluster().getMembers().iterator().next().localMember()) {
    // Run restricted method
   }