英文:
kafka streams | switch to different cluster
问题
我有一个需求，我必须根据某个条件将连接到kafka-cluster-1的kafka流切换到kafka-cluster-2。有没有办法实现这个目标？
英文:
I have a requirement where I have to switch the kafka streams which is connected to kafka-cluster-1 to kafka-cluster-2 based on a criteria. Is there a way to achieve this?
答案1
得分: 1
Kafka Streams 仅限于一个集群。
如果您更改引导服务器，所有内部主题状态将丢失，您将需要重新创建您的输入主题。
英文:
Kafka Steams is limited to one cluster.
If you change bootstrap servers, all internal topic state will be lost and you'll need to recreate your input topic(s)
答案2
得分: 0
如果您确实必须这样做，您应该使用MirrorMaker2（或Confluent Replicator）来从cluster-1复制主题（输入、输出和changelog）到cluster-2。但是，您将会有停机时间。工作流程如下：
- 在cluster-2中创建与输入主题相同名称的主题，并配置生产者开始向cluster-2生产数据。
- 一旦cluster-1中的Kafka Stream集群处于空闲状态（已追赶上），则终止它。
- 使用MirrorMaker2将changelog和输出主题复制到cluster-2。
- 一旦changelog和输出主题完全复制，然后您可以重新启动cluster-2中的Kafka Streams集群（请勿使用相同的磁盘，否则检查点偏移将混乱）。
- 一旦Kafka Streams集群从changelog中恢复了其状态，它将开始再次进行处理。
请注意确保主题名称相同 - 如果在复制的集群中添加了前缀/后缀，您将需要启动生产者/消费者来将数据复制到具有完全相同名称的主题中。
英文:
If you absolutely must do so, you should use MirrorMaker2 (or Confluent Replicator) to replicate the topics (input,output, and changelog) from the cluster-1 to cluster-2. But you'll have downtime. The workflow is as follows:
- Create topics with the same name as the input topics in cluster-2 and configure producers to start producing to cluster-2.
- Once the Kafka Stream cluster in cluster-1 is idle (caught up), terminate it.
- User MirrorMaker2 to replicate the changelog and output topics to cluster-2.
- Once the changelog and output topics are fully replicated, then you can restart your Kafka Streams cluster in cluster-2 (do NOT use the same disks, otherwise, the checkpoint offset will be messed up).
- Once the Kafka Streams cluster has restored its state from the changelog, then it will start processing again.
Be careful that the topic names are identical—if there is a prefix/suffix added in the replicated cluster, you'll have to spin up a producer/consumer to copy the data over into a topic with the exact same name.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论