英文:
How to check if a consumer group has fully consumed a topic in Kafka
问题
如果没有现成的工具或应用程序可以做到，我对如何检查消费者组的所有主题是否已完全消费感兴趣。
实现这样一个解决方案的最佳途径是什么？
在Kafka生态系统中是否有工具可以帮助完成这个任务？
英文:
if there is not a util already or application that does it, i would be interested in recommendation on how to check that all the topics of a consumer group have been fully consumed?
What is the best route to implement such a solution ?
Are there utility in the kafka eco-system that would help with that task ?
答案1
得分: 1
如评论中提到的，您希望查看消费者滞后，这可以通过 kafka-consumer-groups --describe 输出来显示。所有零滞后表示消费者已提交了末尾偏移量，和/或您的消费者正在处理快于任何生产者发布到这些主题的情况（这并不总是正确的，所以滞后可能永远不会真正达到零）。
在每个消费者运行时，您可以检查轮询是否返回零记录，或者在 librdkafka API 中，当您达到末尾时，消费者会抛出一个 "分区末尾" 的代码。然后，您需要为每个主题的所有分区以及按 group.id 进行信息聚合。
您可以查看 Burrow 的 HTTP API 是否包含您正在寻找的信息。
英文:
As mentioned in the comments, you want consumer lag, as shown by kafka-consumer-groups --describe output. All lags of zero mean the consumers have committed the end-offset and/or your consumers are processing faster than any producer to those topics (which is not always true, so lag may never truly reach zero)
While each consumer is running, you could check that polling has returned zero records, or in the librdkafka API, the consumer throws an "end of partition" code when you've reached the end. Then you would need to aggregate this information for all partitions of each topic, as well as by group.id.
You could see if the Burrow HTTP API has the information you are looking for.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论