英文:
firestore query issues with 500 concurrent clients
问题
我正在对我的项目进行一些压力测试，并需要你的帮助来理解其行为。我有一个 Web 服务器，它接受用户的 JSON 数据并将其存储在 Firestore 集合中。用户可以查询这些数据。文档 JSON 只有两个字段 id1 和 id2，都是字符串。现在作为压力测试的一部分，我启动了 500 个线程来模拟 500 个客户端，这些客户端查询集合以便每个线程获取 id1 == thread_id 的文档，代码如下：
query := client.Collection("mycollection").Where("id1", "==", my_id)
iter := query.Documents(ctx)
snapList, err = iter.GetAll()
我遇到了两个问题：
- 一些查询需要很长时间，长达 20 秒才返回结果。
- 一些查询失败，显示连接错误或 I/O 超时。我使用的是 Go SDK。
> "message":"error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp x.x.x.x:443: i/o timeout"}
根据 Firestore 文档，最多允许 100 万个并发客户端。那么为什么只有 500 个客户端就出现问题呢？即使在空集合上运行此测试，我也观察到相同的行为。我是否遗漏了其他的速率限制？
英文:
I am doing some stress testing on my project and need your help in understanding the behavior. I have a web server which accepts json data from users and stores it in a firestore collection. Users can query this data. Document json only has two fields id1 and id2 and both are strings. Now as part of my stress test I start 500 threads to mimic 500 clients which query the collection to give each thread the documents where id1 == thread_id like this:
query := client.Collection("mycollection").Where("id1", ==, my_id)
iter := query.Documents(ctx)
snapList, err = iter.GetAll()
I see two issues:
- some of these queries are taking so long upto 20 seconds to return.
- Some of the queries fail with connection error/io time out. I am using go sdk.
> "message":"error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp x.x.x.x:443: i/o timeout""}
as per firestore documentation uptop 1 million concurrent clients are allowed. Then why am I getting issues on just 500? Even running this test on an empty collection I observe the same behavior. Is there any other rate limit that I am missing?
New
答案1
得分: 1
在向Firestore添加负载时，建议遵循500/50/5规则，该规则在ramping up traffic的文档中有解释：
为了给Cloud Firestore足够的时间准备增加的流量所需的文档，您应该逐渐增加对新集合或词典顺序接近的文档的流量。我们建议从每秒最多500个操作开始到新集合，然后每5分钟增加50%的流量。您可以类似地增加写入流量，但请记住Cloud Firestore标准限制。确保操作在关键范围内相对均匀地分布。这被称为"500/50/5"规则。
因此，您可能希望从较低数量的线程开始，然后每5分钟增加50%，直到达到所需的负载。一些SDK甚至提供了支持类，例如Node.js中的BulkWriter。
英文:
When adding load to Firestore it is recommended to follow the 500/50/5 rule, as explained in the documentation on ramping up traffic:
> You should gradually ramp up traffic to new collections or lexicographically close documents to give Cloud Firestore sufficient time to prepare documents for increased traffic. We recommend starting with a maximum of 500 operations per second to a new collection and then increasing traffic by 50% every 5 minutes. You can similarly ramp up your write traffic, but keep in mind the Cloud Firestore Standard Limits. Be sure that operations are distributed relatively evenly throughout the key range. This is called the "500/50/5" rule.
So you might want to start with a lower number of threads, and then increase 50% every 5 minutes until you reach the desired load. Some of the SDKs even have support classes for this, such as the BulkWriter in Node.js.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论