英文:
Subscribe to every topic partition
问题
什么是订阅给定Kafka主题多次的规范方法，并且对于每个KafkaConsumer从每个分区接收每条消息。
我目前正在做的是生成一个随机的Uuid group.id，以便每个订阅都是一个新的组，但是由于这些订阅的生命周期很短（而且数量很多），Kafka存储有关它们的元数据的开销可能是有害的。
实现这一目标的正确方法是什么？
英文:
What is the canonical way to subscribe multiple times to a given Kafka topic and receive every message from every partition for each KafkaConsumer.
What I am doing as the moment is generating a random Uuid group.id so that each subscription is a new group, but given these subscriptions are short-lived (and there are many of them), the overhead of Kafka storing metadata about them might be detrimental.
What is the correct way to acheive this?
答案1
得分: 1
我认为这个问题的答案是使用 assign() 方法，而不是 subscribe()。
通过这种方法进行手动主题分配不会使用消费者组管理功能。
参考链接：https://kafka.apache.org/26/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
英文:
I believe the answer to this question is to use the assign() method rather than subscribe().
> Manual topic assignment through this method does not use the
> consumer's group management functionality.
Reference: https://kafka.apache.org/26/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
答案2
得分: 0
好的，以下是已翻译的内容：
确保您的消费者组是唯一的，这是确保在组内运行的消费者订阅所有分区并接收所有消息的方法。这就是多个消费者组订阅相同主题的目的。
我同意，这需要您创建多个消费者组，这会增加元数据开销。但这完全取决于您的用例要求，无论您是否需要单个/多个消费者组。
英文:
Well, having unique consumer groups is the way to ensure that your consumer(s) running inside a group subscribes to all partitions and receives all the messages. That is the purpose of multiple consumer group subscribing to the same topic.
I agree that it requires you to create multiple consumer groups and that gives the overhead of metadata. But it all depends on your usecase requirement whether you want single/multiple consumer groups.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论