出现了一段时间内获取了太多的频道错误。

huangapple go评论64阅读模式
英文:

Getting too many channels error for a period of time

问题

I have 16 queues & multiple consumer servers for those queues. I have created one dedicated channel for each queue to consume messages. Consumer & dispatch channels on each server share same connection.


When I dispatch messages to each queue, I do the following:

  1. create a new channel
  2. bind channel to the queue with proper routing
  3. dispatch the message
  4. close the channel

I have lots of incoming webhooks from Shopify & these webooks contents are dispatched to specific queues.

While processing each message, I need make an API call to Shopify. Shopify API has rate limit. If I hit rate limit once, I redispatch all messages from the consumer back to rabbitmq with a delay header of 1 minute (time required to clear the API rate limit). Now, when I have several consumers running with lots of messages in the queue & I re-dispatch those messages, I get too many channels error for a period of time. How can I avoid this error?


I tried to keep 2 dedicated channels per queue:

  1. for consumer purpose only
  2. for dispatch purpose only

For 16 queues, & around 11 consumer servers. This way, I always have to keep 352 channels open. This causes CPU utilization on rabbitmq host server to reach >90% which is also an issue. As the server can crash any time.

英文:

I have 16 queues & mulitple consumer servers for those queues. I have created one dedicated channel for each queue to consume messages. Consumer & dispatch channels on each server share same connection.


When I dispatch messages to each queue, I do the following:

  1. create a new channel
  2. bind channel to the queue with proper routing
  3. dispatch the message
  4. close the channel

I have lots of incoming webhooks from Shopify & these webooks contents are dispatched to specific queues.

> While processing each message, I need make an API call to Shopify. Shopify API has rate limit. If I hit rate limit once, I redispatch all messages from the consumer back to rabbitmq with a delay header of 1 minute(time required to clear the API rate limit).

Now, when I have several consumers running with lots of messages in the queue & I re-dispatch those messages, I get too many channels error for a period of time. How can I avoid this error?


I tried to keep 2 dedicated channels per queue:

  1. for conusmer purpose only
  2. for dispatch purpose only

For, 16 queues, & around 11 consumer servers. This way, I always have to keep 352 channel open. This caues CPU utilization on rabbitmq host server to reach >90% which is also an issue. As the server can crash any time.

答案1

得分: 1

我在查阅RabbitMQ文档后找到了解决问题的方法。

与其为每次分发创建新通道,我创建了一个单一通道并且在整个连接会话期间保持它活动。在创建通道时,我声明了所有我的队列将使用的交换机

然后,我只需将消息发布到所需的交换机上,使用路由键。由于我的队列已经与交换机绑定,并监听具有特定路由键的消息,因此消息最终进入了正确的队列!

这样,我可以仅维护01个连接和每个服务器只有01个通道!

英文:

I found the solution to the problem after digging through the RabbitMQ documentation.

Instead of creating a new channel for each dispatch, I created a single channel & kept it alive for the entire connection session. When creating the channel, I asserted all the exchanges that would be used by my queues.

Then I just publish the messages to the desired exchange with the routing key. As my queues are already bonded with the exchanges & listen for messages with a given routing key, the messages end up in the correct queue!

This way I can maintain just 01 connection & only 01 channel per server!

huangapple
  • 本文由 发表于 2023年1月10日 14:54:01
  • 转载请务必保留本文链接:https://go.coder-hub.com/75066431.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定