英文:
How do you solve concurrency issues when horizontally scaling microservices that are sharing a DB?
问题
让我们假设您正在运行两个共享MongoDB数据库的Tickets Service实例。
我们从客户端收到了对同一张票的两个更新,这些更新将由Tickets Service实例通过负载平衡并发处理。由于某种原因,在另一个更新能够到达数据库之前,另一个更新已经更新了票,从而破坏了请求的原始顺序。因此,应该在之前处理的请求被丢弃(例如)当采用乐观并发控制版本控制系统并创建可能对某些数据(如帐户余额)造成潜在危险的不一致性。
如何解决这种问题并确保正确的排序和一致性?
英文:
Let's assume that you are running two instances of a Tickets Service sharing a MongoDB database.
We receive two updates on the same ticket from a client that will be handled concurrently by the Tickets Service instances thanks to load balancing. For some reason, the request made after the other one updates the ticket before the other could even reach the database thus breaking the original order of the requests. So, the request that should have been handled before is discarded (for example) when adopting an Optimistic Concurrency Control versioning system and creates an inconsistency that could potentially be dangerous for some kind data (like account balances).
How do you solve this kind of problem and guarantee the correct ordering and consistency?
答案1
得分: 1
如果您使用linearizable
读关注点与majority
写关注点,执行查询的线程将看起来像是一个单一线程执行操作。如果您使用replaceOne
查询,似乎可以像Guru的答案中那样实现乐观并发控制:使用linearizable
读关注点读取文档(在文档中包含版本,例如"version": 42
),构建一个具有增加版本的新文档(例如"version": 43
),然后使用majority
写关注点的replaceOne
来确保只有在大多数节点仍然看到先前版本时才会将文档更新到此版本。如果写操作失败,因为大多数节点具有较新的版本,重新读取文档。
Mongo文档指出,这通常不如其他策略高效,尤其是在需要多次尝试的情况下。如果期望有相当高的并发性,可能值得让您的微服务实例在彼此之间形成有状态的集群,并在它们之间分配操作特定票据或帐户的责任。由于一个实例负责特定的票据/帐户等,可以使用本地悲观并发控制来有效地使对Mongo的操作(包括“修改”部分的“读取-修改-写入”)在给定文档上由单一线程执行:这也可以让您安全地将值缓存到实例的内存中(将“读取-修改-写入-读取-修改-写入...”变成“读取-修改-写入-修改-写入...”),具体取决于修改尝试的频率,这可能是一个重大胜利或更多的额外复杂性。这在使用actor风格方法时特别常见,其中有工具来简化所有这些操作(例如golang的Service Weaver、Akka.Net、JVM上的Akka、Orleans或Erlang/Elixir...免责声明:我的雇主维护其中一个项目)。
英文:
If you use the linearizable
read concern with the majority
write concern, the threads performing queries will appear to be executing queries as if a single thread executed the operations. If you use the replaceOne
query, it looks like it should be possible to implement optimistic concurrency control as in Guru's answer: read a document with linearizable
read concern (include a version in the document, e.g. "version": 42
), construct a new document with an incremented version (e.g. "version": 43
), and use replaceOne
with majority
write concern to ensure that the document will only be updated to this version if a majority of nodes still see the previous version. If the write fails because a majority of nodes have a later version, re-read the document.
The Mongo docs note that this will generally not be as performant as other strategies, especially in the case where you have to attempt things multiple times. If a fairly high level of concurrency is expected, it may be worth having your microservice instances form a stateful cluster among themselves and allocate responsibility for operating on a particular ticket or account amongst them. Since one instance is responsible for a given ticket/account/etc., local pessimistic concurrency control can be used to effectively make the operations (including the "modify" part of "read-modify-write") against Mongo on a given document be executed by a single thread: this also can let you safely cache values in the instance's memory (turning "read-modify-write-read-modify-write..." into "read-modify-write-modify-write..."): depending on how frequently the modifications are being attempted, this may be a major win or a lot of extra complexity. It's especially common in actor style approaches where there's tooling to simplify a lot of this (e.g. Service Weaver for golang, Akka.Net, Akka on the JVM, Orleans, or Erlang/Elixir... disclaimer: my employer maintains one of those projects).
答案2
得分: 0
> In my case the two requests should be saved into the database with the same order made by the client
在我的情况下,两个请求应该按照客户端的顺序保存到数据库中。
> however due to the fact that the Tickets Service have two instances, the order is being altered.
然而,由于票务服务有两个实例,顺序被改变了。
> Consider the case when an user wants to deposit and withdraw from his account balance, you must respect the order of the requests made by the client.
考虑一下当用户想要从他的账户余额中存款和取款时,您必须尊重客户端发出的请求的顺序。
TL;DR
简而言之,在某些情况下,您可能需要严格要求客户端顺序,但在大多数("企业")系统中(至少我遇到的情况如此),通常您关心的是在服务器上强制一些顺序,而您可以根据您的基础架构使用几种技术来实现这一点。
英文:
> In my case the two requests should be saved into the database with the same order made by the client
The almost only way to somewhat guarantee this I can think of is to guarantee that user can uses only single client instance (for example via blocking login which will validate there are no other active sessions) at a time and make that client instance assign the ordering to the send requests (i.e. atomically increment and use some counter client side which will be then used server side for correct ordering), which obviously not something you usually want in some accounting system (for example if you are writing some banking system you usually do not want to prevent user from using ATM while using mobile bank client).
> however due to the fact that the Tickets Service have two instances, the order is being altered.
If we are talking about using such communication channels like HTTP, TCP/IP, etc. you can't actually guarantee the ordering even if you have a single service instance because the order can be scrambled long before the requests hit your server (due to transport specifics or even client machine CPU scheduling, in theory), not to mention that single instance still usually processes requests in parallel (though here could be nuances) and is susceptible in general to the same issues but on smaller scale (and can use some other tools for synchronization).
> Consider the case when an user wants to deposit and withdraw from his account balance, you must respect the order of the requests made by the client.
Actually you kind of don't. You must respect business rules like you can't withdraw more than there is on account balance + allowed overdraft. If user sends withdraw request which will overdraft the balance over the limit before the deposit request is processed or even acknowledged then it is kind of client problem and the withdraw attempt should be retried.
What (I would argue) you actually want/need is to guarantee that your two instances will not perform "unatomic"/non-synchronized updates on the same data. Usually it is handled via transactions on the database side with appropriate isolation levels. Another approach can be to test for optimistic concurrency violation approach, i.e. (for most relational databases) you can just use query looking something like the following:
update Ticket
set Version = new_unique_id -- for example guid, or next id from sequence
, ... -- rest of the update
where Id = ... and Version = current_unique_id
And then check if the returned number of updated rows is equal to 1.
TL;DR
There are cases when you can require strict client ordering but in majority of ("enterprise") systems (at least which I have encountered) do not, usually you care about enforcing some ordering on server and there are several techniques for that which you can use depending on your infrastructure.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论