英文:
Managing maximum delivery attempts and rate limiting separately with Google Cloud Pub/Sub
问题
我正在使用Google Cloud Pub/Sub提供的最大交付尝试功能。此功能允许在五次交付失败尝试后将消息发送到死信主题，这对于在发生故障时进行手动消息处理非常有益。
然而，我们还通过设置Max-instances=2在我们的函数上实施了人工速率限制。目标是始终有两个函数以最大容量对API进行ping。
我的期望是死信主题只会收到失败的消息。不幸的是，它还接收到无法按时交付的消息，导致429错误："请求因没有可用实例而中止。"这些消息可能会被多次重试，取决于实例的繁忙状态，可能会达到20次、30次甚至40次。
将最大交付尝试设置为40不是理想的，因为这意味着要重试40次失败。
是否有一种分别处理故障和限制的方法？
谢谢！
Béranger
英文:
I am utilizing the Maximum Delivery Attempts feature provided by Google Cloud Pub/Sub. This feature allows messages to be sent to a dead-letter topic after five failed delivery attempts, which is beneficial for manual message handling in case of failures.
However, we are also implementing an artificial rate limit on our Functions by setting Max-instances=2. The objective is to constantly have two functions pinging the API at maximum capacity.
Maximum delivery attempts: 5
Max instances : 2
My expectation was that the dead-letter topic would only receive failed messages. Unfortunately, it is also receiving messages that couldn't be delivered on time, resulting in a 429 error: "The request was aborted because there was no available instance." These messages might be retried multiple times, potentially 20, 30, or even 40 times, depending on the busy state of the instance.
Setting the Maximum Delivery Attempts to 40 is not ideal since it would mean retrying a failure 40 times.
Is there a way to handle failures and limiting separately?
Thank you!
--
Béranger
答案1
得分: 2
以下是已翻译的内容：
你无法更改 PubSub Push 消息的错误处理方式。
PubSub 向您的服务发出的请求的每个响应，如果不是 1xx/2xx（请参阅 https://cloud.google.com/pubsub/docs/push?hl=en#receive_push），都将被视为失败，从而触发重试。
您需要限制自己的服务速率，还是因为调用其他服务而需要限制速率？
如果是后者，您可以在您的服务中实现速率限制。有许多库可用于此。
否则，您可以通过立即确认超出速率的消息而不处理它们来实施速率限制，然后您的服务只需将相同的消息重新发布到源主题。
编辑：我假设您以某种方式知道是否达到了限制。如果是这种情况，您可以在每条传入消息上检查是否可以调用外部 API。如果可以，您就处理该消息。如果已达到限制，您可以要么只返回 2xx 而不进行进一步处理（= 丢弃该消息），要么重新发布相同的消息然后返回 2xx（= 手动重放/重试）该消息。这样，"有效" 消息就不会触发重试限制，因为您生成了新消息（具有相同内容）。
英文:
You cannot change the error handling for PubSub Push messages.
Every answer to the request from PubSub to your service whichs is not 1xx/2xx (see https://cloud.google.com/pubsub/docs/push?hl=en#receive_push) will be handled as a failure which leads to a retry.
Do you need to rate limit your own service or do you need to rate limit because of another service you call yourself?
If it is the latter, you can implement the rate limiting in your services. There are plenty libraries for this.
Otherwise you could implement rate limiting by acknowledging the out-of-rate messages immediately without processing them - and then your service just publishes the same message back to the source topic.
Edit: I assume that you somehow know if you reached your limit or not. If this is the case, you check on each incoming message if you may call the external API or not. If so, you process the message. If you reached your limit, you either just return 2xx without furter processing (= discarding the message) or you publish the same message again and then return 2xx ( = manually replaying / retrying) the message. this way, "valid" messages do not hit the retry limit as you produce new messages (with the same content)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论