英文:
Kafka consumers in consumer group reading from different Kafka cluster
问题
我们有两个Kafka集群，分别是A和B。我们正在将工作负载从集群A迁移到集群B。在执行此迁移时，有一段时间我们希望消费者可以同时从两个集群中读取消息，这些消费者属于同一消费者组。
示例：
在集群A中有topic-1，在集群B中也有topic-1。我们有一个名为topic-1-cg的消费者组。在迁移过程中，我们将在集群B中实例化具有相同group-id，即topic-1-cg的消费者。这将使消费者组中的消费者指向两个集群，即一些消费者从集群A读取消息，而一些从集群B读取消息。这两个消费者都使用相同的消费者组ID。
我的问题是，是否允许使用这种消费者配置？我无法在本地模拟多集群设置。
英文:
We have two Kafka clusters A and B. We are migrating the workload from cluster A to cluster B. While performing this migration at some point we want consumer to read from both the cluster being part of same consumer-group.
Example:
There is topic-1 in cluster A and topic-1 in cluster B. We have consumer group called topic-1-cg. During migration we will instantiate consumers in cluster B with same group-id i.e. topic-1-cg. This will make the consumer group having consumers pointing to both clusters i.e. few consumers are reading from cluster A and few from cluster B. Both consumers are instantiated with same consumer-group ID.
My question is, is this consumer configuration allowed to use? I am not able to simulate multi-cluster setup locally.
答案1
得分: 2
不，这种设置不会导致消费者从两个集群中读取。
当你有两个不同的集群时，它们不会互相通信，这就是为什么它们不知道在不同的消费者组中是否使用了相同的消费者组名称。
因此，即使在两个不同的集群中使用相同的主题名称和相同的消费者组名称，它们也不会导致从两个集群中进行消费。
英文:
No, this setup with not result in consumers reading from both the clusters.
When you have 2 different clusters, they do not talk to each other and that is why they don't know if the same consumer group name is being used in a different consumer group.
So even if you have same topic name and use same consumer group name in 2 different clusters, they will not result in consuming from both clusters.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论