英文:
What is the difference between appengine datastore timeout errors 5 and 11?
问题
我正在尝试加速一个Google App Engine的请求处理程序,其中包含一个大的数据存储PutMulti
调用(500个实体),通过将其拆分为实体的批次,并运行并发的goroutine来发送较小的PutMulti
调用(每个调用100个实体)。
在此之前,当我在许多并发请求上测试处理程序时,我经常收到数据存储错误Call error 11: Deadline exceeded (timeout)
,这是由于我的PutMulti
调用超过了截止时间。并行化之后,处理程序的速度确实加快了,但我偶尔仍然会遇到该错误,以及另一种类型的错误,即API error 5 (datastore_v3: TIMEOUT): The datastore operation timed out, or the data was temporarily unavailable
。
这个错误5是由于数据存储中的争用引起的吗?错误5和错误11之间有什么区别?
英文:
I'm trying to speed up a Google App Engine request handler that has a big datastore PutMulti
call (500 entities) by splitting it into batches of entities and running concurrent goroutines to send smallerPutMulti
calls (100 entities each).
Before this, I had often been getting the datastore error Call error 11: Deadline exceeded (timeout)
from my PutMulti
calls going over the deadline when I tested the handler on many concurrent requests. After the parallelization, the handler did speed up, but I still occasionally got that error and also another type of error, API error 5 (datastore_v3: TIMEOUT): The datastore operation timed out, or the data was temporarily unavailable
.
Is this error 5 due to contention in the datastore, and what is the difference between errors 5 and 11?
答案1
得分: 3
这些错误来自两个不同的地方,第一个是调用错误,是由RPC客户端的超时引起的本地错误。它表示等待RPC完成的超时。google.golang.org/appengine中默认的RPC超时时间是60秒。
第二个错误来自服务端。这个错误表示在数据存储中执行操作时发生了超时。其中一些操作的超时时间要比60秒短得多,通常这可能表示争用。
更简单地理解这些差异的一种方式是,如果你进行一个包含大量更改的单个多操作,你很容易触发第一个超时。如果你对单个键或小组键进行大量并发操作,你更容易触发后者。由于超时是共享资源饱和的一般指标,当然有很多方法和组合可以生成它们。一般来说,你会希望根据需要重试操作,并适当调整操作的大小,尽可能地将操作聚合在热键上,以减少与争用相关的问题的机会。正如其他人建议的那样,Python和Java文档中已经有一些示例。
你可以使用https://godoc.org/google.golang.org/appengine#IsTimeoutError,如果你需要增加第一个错误类的超时时间,你可以调整上下文截止时间,参见这里的方法:https://godoc.org/golang.org/x/net/context#WithDeadline 注意:你将无法将截止时间延长到请求截止时间之后,但是,如果你在任务或虚拟机中运行,你可以延长到较长的截止时间。
英文:
These errors come from two different places, the first, the call error, is a local error that is caused by a timeout in the RPC client. It indicates that there was a timeout waiting for completion of an RPC. The default RPC timeout in google.golang.org/appengine is 60 seconds.
The second error comes from the service side. This error indicates that a timeout occurred performing operations within datastore. Some of these operations have timeouts much shorter than 60s, and typically this may indicate contention.
A possibly simpler way to understand the differences is that you will find that if you make a single multi operation with a very large number of changes, you can trigger the first timeout with ease. If you create a significant number of concurrent operations against a single key or small set of keys, you will more readily trigger the latter. As timeouts are general indicators of saturation of shared resources, there are of course many ways and combinations to generate them. In general, one will want to retry operations as appropriate, and also size operations appropriately, as well as aggregating operations on hot keys as best as possible to reduce the chance of contention related issues. As others have suggested, the python and java docs have some examples of this already.
You may wish to make use of https://godoc.org/google.golang.org/appengine#IsTimeoutError and if you need to increase the timeout for the first error class, you may be able to adjust the context deadline, see the methods here: https://godoc.org/golang.org/x/net/context#WithDeadline Note: you will not be able to extend the deadline beyond that of a request deadline, however, if you are running in tasks or VMs you can extend to long deadlines.
答案2
得分: 1
你看到的第一个错误可能只是正常操作中的超时问题,第二个错误可能是由于写入冲突引起的。更多信息请参考:处理 Datastore 错误 https://cloud.google.com/appengine/articles/handling_datastore_errors
英文:
The first error you see may be just the timeout in normal operation, the 2nd is likely because of write contention. More on this: Handling Datastore Errors https://cloud.google.com/appengine/articles/handling_datastore_errors
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论