英文:
Golang delete all records from a topic in Kafka
问题
嗨，我正在使用Go和Kafka编写一个服务，并且需要实现一个删除所有记录的端点，该端点将从特定的主题中删除所有记录。然而，我找不到一个合适的方法来做到这一点。我正在使用Sarama库来处理Kafka。
到目前为止，我找到的唯一两种实现删除所有记录的方法是删除主题，但这似乎不是处理这个问题的高效方式；第二种方法是使用Sarama库中的DeleteRecords函数，但是该函数只删除偏移量小于相应分区的给定偏移量的记录。这意味着我必须先获取最新的偏移量。
基本上，我正在寻找做这件事的最佳方法。有人可以帮助我吗？有什么最佳实践吗？也许我漏掉了什么。非常感谢提供一些示例。谢谢！
英文:
Hi I am writing a service in Go and Kafka and I need to implement a delete all endpoint which would delete all records from a specific topic. However I can not find a proper way to do that. I am using the Sarama library for Kafka.
So far the only two ways I can find to implement delete all is by deleting the topic which does not seem to be an efficient way to handle this problem and the second one is using the DeleteRecords function from the Sarama library, however this function Deletes records whose offset is smaller than the given offset of the corresponding partition. Which means that I have to get the latest offset first.
Basically I am looking for the best way to do such a thing. Could anyone help me? What are the best practices? Maybe I have missed something. I would really appreciate some examples. Thank you!
答案1
得分: 0
如果您想要修剪所有消息，另一种方法是将主题的保留时间缩短到一个较小的值（例如100毫秒）。等待代理服务器从主题中删除所有记录，然后将主题的保留时间恢复为原始值。以下是如何操作的步骤：
首先，将保留时间设置为100毫秒。
kafka-configs --zookeeper localhost:2181 \
--entity-type topics \
--entity-name my-topic \
--alter --add-config retention.ms=100
替代解决方案：
删除主题并重新创建
虽然不如前两种方法优雅，但在某些情况下可能是更简单的解决方案（例如，如果主题创建是脚本化的）。
kafka-topics --bootstrap-server localhost:9092 \
--topic my-topic \
--delete
然后重新创建主题：
kafka-topics --bootstrap-server localhost:9092 \
--topic my-topic \
--create \
--partitions <number_of_partitions> \
--replication-factor <replication_factor>
英文:
If you want to prune all the messages, another way to do that is to reduce the retention of the topic to a small value (e.g. 100ms). Wait for the brokers to remove all the records from the topic and then set the topic retention to its original value. Here’s how to do it.
First, set the retention time to 100 milliseconds.
kafka-configs --zookeeper localhost:2181 \
--entity-type topics \
--entity-name my-topic \
--alter --add-config retention.ms=100
Alternative Solution:
Delete a topic and create it again
Not as elegant as the previous two approaches, yet it might be an easier solution in some cases (e.g. if topic creation is scripted).
kafka-topics --bootstrap-server localhost:9092 \
--topic my-topic \
--delete
Then create it again:
kafka-topics --bootstrap-server localhost:9092 \
--topic my-topic \
--create \
--partitions <number_of_partitions> \
--replication-factor <replication_factor>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论