Event Sourcing and CQRS: Handle concurrency with multiple "read model consumers" instances

I'm implementing a solution based on Event Sourcing and CQRS patterns.

In my use case I have:

• A WRITE microservice: Where events are appended to a particular Stream (let's call it X Stream) stored on an EventStoreDB instance.
• A READ microservice: Subscribed to the X Stream of the EventStoreDB, which consumes the appended events and store the projected model in a MongoDB instance.

In a simple scenario, where there is a single instance of the READ microservice, all works as expected:

1. An event is appended to the X Stream in the EventStoreDB
2. The single instance of the READ microservice consumes the event and stores the projected model on the MongoDB instance

Now suppose that you want to scale out the READ microservice (the event's consumer) to two or more instances. This is what will happen:

1. An event is appended to the X Stream in the EventStoreDB
2. Each replica of the READ microservice consumes the event and tries to store the projected model on the MongoDB instances, corrupting the READ model (because of the concurrent write).

Is there a way to handle this scenario?

Usually there is 1 active process with the catch-up subscription doing update to your read model.

And eventually a second one on stand-by if that first process should stop unexpectedly

When that becomes too slow , you can have multiple processes and partition them in such a way that they would handle a specific set of documents on the same target store.

Yves wrote the correct answer, I just want to add a couple of things.

I could add that write and read side scaling models are completely different.

• The write side scales seamlessly without many constraints except how many read-write operations the database can tolerate (depends on the instance size).
• The read side is constrained by performance of the target database. Obviously, running a projector linearly in a single subscription instance will hit the physical limitation of how many round trips to the target database you can do in a given period of time (say, one sec).
• The read side scalability also depends on the ordering requirement. If you need events to be ordered in the whole log, or a category - it's one thing. If you only care about events from a single stream being projected in order, it's different. The ordering requirement gives you the idea about how you can partition the read model updates.

I made projections a lot faster by applying partitioning by stream, but it still is a single process. In many cases it is fine, as it can project thousands events per minute. The high availability concern is legit for the purpose of increased redundancy in case of failures, but then again, applying simple health checks as prevention measures will ensure that the subscription workload gets restarted if it is stuck.

We are baking something that might remove the whole issue all together, but I won't dare mentioning any dates as we are still researching the topic.

"Now suppose that you want to scale out the READ microservice (the event's consumer) to two or more instances. This is what will happen:

An event is appended to the X Stream in the EventStoreDB
Each replica of the READ microservice consumes the event and tries to store the projected model on the MongoDB instances, corrupting the READ model (because of the concurrent write).
Is there a way to handle this scenario?"

Yes run with four distinct read models not one.

I know that sounds blatantly obvious but many miss it. Run four distinct instances of mongodb not one cluster and four sets putting into it.

Still thinking back about this use case, I'm wondering if implementing idempotent listeners could be an elegant idea:

*For example, one might use the Event ID as a correlation-id in the Read Model to ensure the idempotency of a listener.

What do you think about this approach?