Sharded concurrency for Cloud Tasks

Google Cloud Tasks queues have a maxConcurrentDispatches parameter that specifies the number of tasks that can execute concurrently for a queue.

Now let's say I have an operation is performed on a user, and any number (with an upper limit) of operations can execute at the same time, but only one at a time per user (e.g. to reduce database contention).

As far as I can imagine, I have two options

1. Specify maxConcurrentDispatches: 1 so that only one operation can run at a time. However with many thousands of users, this will not scale, especially if it is a long-running operation. The operation throughput will be too slow.
2. Create many different queues, one for each user, each with maxConcurrentDispatches: 1, so that any number can execute at the same time but one for each user. However I'm concerned that this too will not scale well. It will become a burden to maintain hundreds of thousands if not millions of queues (what happens if I need to pause all the queues, or change their configuration?). Also I cannot set an upper bound on overall concurrency. Can Cloud Tasks even support a potentially unlimited number of queues?

How should I implement this scenario on Google Cloud Platform? Is there another messaging service that could use, or a design decision that I am overlooking?

You could use PubSub instead of Cloud Task. Configure the HTTP push subscription to perform the call to the endpoint (as Cloud Task do today).

The trick here is to use an ordering key. Set the UserID as ordering key and like that the next message for this userID won't be delivered until the current one has not been acknowledged.

Many message will be delivered in the same time as long as they don't have the same ordering key

The "problem" (if it's one): the message will be unqueued in the same order as they are created (FIFO)

You can specify name attribute when creating a task, and submitting task with identical name again will fail with ALREADY_EXISTS error. So if you will use ID of the user in the task name, you can roughly get one task per user.

This has however a few disadvantages. When you specify the name, the tasks dispatching might have increased latency (because duplicate names need to be checked). And you cant reuse the task name for some time after the previous task has finished successfully. I dont know the exact time, but it's in scale of minutes, please check the docs.

Workaround for the latter issue could be some kind of counter for each user and add it to the task name, or maybe use some timestamp (ie. if user can send one task a minute then use combination of userID + currentMinuteTimestamp()?