英文:
what is the maximum tasks for kafka streams on multiple topcis with different partition
问题
- 假设我们有两个输入主题。Topic1有2个分区，Topic2有4个分区。
- 我们使用1个线程创建了Kafka流应用程序。
问题：我们可以运行多少个流应用程序，以便每个都被分配一个分区？
根据我的理解，这取决于输入主题的最大分区数，即4个。
但是，我想要实现的是6个，即所有主题分区数的总和。您知道是否可行吗？谢谢。
英文:
- Suppose we have two input topics. Topic1 has 2 partitions, and Topic2 has 4 partitions.
- We create the kafka stream application with thread number 1.
Question: what is the maximum number that we can run the stream application that all will be assigned a partition?
as for my understanding, it is decided by the maximum partition of input topics. that is 4.
while what I want to achieve is 6, that is the sum number of all topics' partition. do you know is this doable? Thanks.
答案1
得分: 1
流应用程序的并行性是由输入主题中的分区数定义的，你说得对。这是无法更改的。一种解决方法是使用一个中间的重新分区主题：你可以将输入主题重新分区成一个具有6个分区的新主题，然后使用并行度为6来执行实际工作。
英文:
The parallelism of a streams application is defined number of partitions in the input topic(s), you are correct. You cannot change this. A workaround would be to work with an intermediate repartition topic: you repartition the input topic into a new topic with 6 partitions, and then do the actual work with a parallelism of 6.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论