英文:
Ideal number of Kafka topics per Kafka cluster
问题
我们有一个使用案例，我们希望有大约2000个Kafka主题（我们有3个代理），每个主题有1个分区。关于一个集群可以有的主题的最大数量，我们有没有已知的建议？我浏览了以下网址，但它是关于分区而不是主题的数量。
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/
如果我们有更多的主题，是否会增加集群管理的额外开销？
我在网上进行了大量搜索，但没有找到与高数量的Kafka主题相关的建议或已知限制。
英文:
We have a use case where we would like to have around 2000 kafka topics (we have 3 brokers) and 1 partition per topic. Do we have any known recommendation on what is the maximum number of topics that one cluster can have? I went through the following url but it is about partitions but not number of topics.
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/
Would there be any additional overhead on management of the cluster if we have more topics?
I have done a lots of searches on web but did not find any recommendations or known limitations w.r.t. high number of kafka topics.
答案1
得分: 1
2000是可以的。是的，对于每个分区（不仅仅是主题），随着分区被复制并写入磁盘，每个分区都会产生更多的开销，这将导致网络和磁盘IO。
不建议为您的集群中的每个主题创建一个分区（不管怎样，消费者偏移主题已经有更多分区了）。
英文:
2000 is fine. Yes, there will be more and more overhead for each partition (not only topic) that's managed, as partitions are replicated and written to disk, causing network and disk IO for each and every one of them.
1 partition for every topic in your cluster is not recommended. (the consumer offsets topic has more than that, anyway)
答案2
得分: -1
for MSK there is limit for partition per broker.
英文:
for MSK there is limit for partition per broker.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论