英文:
Kafka consumers picking up the same message again
问题
我有一个包含30个实例的golang服务。有一个包含3个分区的主题，一个单一的消费者组监听这些消息。我们面临的问题是，每条消息都会被同一个消费者反复接收，每次间隔24-30小时。
我确认消息只被推送到主题一次，而且每次消息都是相同的，但是消息的接收时间间隔不同，例如：
- 2021年6月2日，12:59:29.555
- 2021年6月3日，20:33:11.438
- 2021年6月5日，12:08:40.710
- 2021年6月6日，12:34:39.188
请问有人可以帮忙找出问题所在吗？谢谢。
英文:
I have a golang service with 30 instances. There is a topic with 3 partitions and a single consumer group listening to the messages. The issue we are facing is that each message is being picked up by the same consumer again and again with a gap of 24-30 hours.
I have confirmed that the message is being pushed to the topic once and even the message is same every time but the message is being picked up at different time intervals, e.g.:
- June 2nd 2021, 12:59:29.555
- June 3rd 2021, 20:33:11.438
- June 5th 2021, 12:08:40.710
- June 6th 2021, 12:34:39.188
Can someone please help with what the problem could be. Thanks
答案1
得分: 3
-
你不需要30个实例来消费3个分区。因为分区不会在消费者实例之间共享。所以，如果你的主题有3个分区，你的最大消费者实例数应该是3个。
-
如果你的消费者在消费后没有提交读取的消息，就可能发生重新消费。请查看消费者提交间隔和自动提交是否启用。如果你消费的消息没有被提交，那么在消费者重新平衡之后，这些消息可能会被重新消费。
英文:
-
You don't need 30 instances to consume from 3 partitions. Because partitions are not shared between consumer instances. So, if your topic has 3 partitions, your maximum number of consumer instances should be 3.
-
Re-Consuming can be happen if your consumers not commit the read massage after consumed. Look in to consumer commit interval and auto commit enabled or disabled. If your consumed message is not committed, then after rebalance happened to consumers, those messages can be re-consumed.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论