英文:
nsq go client can't keep up
问题
我是一名Go初学者，来自Node世界，正在使用官方的Bitly Go客户端构建一个消费者。我正在使用AddConcurrentHandlers来生成50个goroutine来处理大量的消息。问题是，我的消费者落后了，在nsq上留下了指数级的未处理/接收的消息。还有其他人遇到过这个问题吗？
我在Node中构建了相同的东西，以查看是否存在服务器或NSQ配置问题，并且它能够快速处理所有消息。
GO代码：
q, _ := nsq.NewConsumer("chat", "golangbetches", config)
q.AddConcurrentHandlers(nsq.HandlerFunc(func(message *nsq.Message) error {
l.Debug("Got a message: %v", message)
message.Finish()
return nil
}), 50)
err := q.ConnectToNSQLookupd("<address here>")
英文:
I'm a Go novice coming from the Node world and I'm building a consumer using the official Bitly Go client. I am using AddConcurrentHandlers to spawn 50 goroutines to handle the fire hose of messages. The issue is that my consumer falls behind leaving an exponential of unprocessed/received messages on nsq. Has anyone else encountered this?
I built the same thing in Node to see if there was a server or NSQ configuration issue and its able to process all messages as quickly as they come in.
GO CODE:
q, _ := nsq.NewConsumer("chat", "golangbetches", config)
q.AddConcurrentHandlers(nsq.HandlerFunc(func(message *nsq.Message) error {
l.Debug("Got a message: %v", message)
message.Finish()
return nil
}), 50)
err := q.ConnectToNSQLookupd("<address here>")
答案1
得分: 4
cfg.MaxInFlight处理的是“此消费者实例允许同时处理的最大消息数...”更多详细信息可以在消费者源代码中找到。
将cfg.MaxInFlight设置为合理的值，因为它的默认值是1，可以在配置文件中找到。
在文档中提供了一个示例配置，可以在这里找到，其中将其设置为1000。这可能适用于您的应用程序，但最好监视它，因为配置错误可能导致消息被截断。
英文:
cfg.MaxInFlight handles the, "maximum number of messages this comsumer instance will allow in-flight..." More details are available in the consumer source
Set cfg.MaxInFlight to something reasonable, as it defaults to 1
An example configuration is available in the documentation where it is set to 1000. This may or may not be suitable for your application; and, you'd do well to monitor it, as a misconfiguration may result in truncated messages.
答案2
得分: 0
另外一种加快Go NSQ消费者速度的方法可以在这里找到：https://github.com/nsqio/go-nsq/issues/187
其中一种方法是增加nsqd上的--max-rdy-count，这样可以将消费者的--max_in_flight增加到默认值2500以上，对我来说有效。
英文:
Another ways to speed up go nsq consumers can be found here: https://github.com/nsqio/go-nsq/issues/187
One of them, which works for me, is increasing --max-rdy-count on nsqd, which allows to increase consumer's --max_in_flight even higher than the default 2500.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论