使用Kafka Go客户端的Consumer,如何倒带并查看分区中的先前偏移量?

huangapple go评论159阅读模式
英文:

How to rewind and look at previous offset in a partition using Kafka Go client's Consumer

问题

我是你的中文翻译助手,以下是翻译好的内容:

我对Kafka还不熟悉。目前我正在尝试使用Confluent Inc的Github仓库中的这个Channel Consumer example进行实验。

据我所知,消费者被分成不同的组。每个组在分区中都有自己的偏移量。假设我在一个特定的主题(我们称之为owner_commands)中有40条消息。一个属于dog组的消费者加入并开始消费这40条消息。

当我断开连接并重新连接这个消费者时,我注意到消息不再显示。它显示已经到达文件末尾。然而,如果我用另一个属于不同组(比如cat)的消费者加入集群,我就可以再次读取那40条消息。

你知道在Kafka的Go API中,有没有一种方法让dog组的消费者倒回并重新播放那些消息?我查看了Kafka Golang API的源代码,但没有找到任何可以让我倒回并查看过去特定消息的指示。

谢谢。

英文:

I am new to Kafka. Currently I am experimenting with this Channel Consumer example from Confluent Inc's Github repo

From what I know, consumers are separated into groups. Each group has their own offset in the partition. Let's say I have 40 messages in a particular topic let's call it, owner_commands. A consumer, belongs to the dog group, joins and begins to consume those 40 messages.

When I disconnected and reconnected this consumer, I noticed that messages don't show up anymore. It says that I have reached the end of file. However, if I join the cluster with another consumer, which belongs to a different group (say cat) I get to read those 40 messages again.

Do you know if there is a way for consumers in the dog group to rewind and replay those messages again using Kafka's Go API. I looked at the source code for Kafka Golang API, I couldn't find anything that indicates to me that I can rewind and look at a particular message in the past.

Thank you

答案1

得分: 3

你可以使用CommitOffsets并将其提交回你想要回滚到的偏移量。下一次轮询将从该偏移量开始。

CommitOffsets的文档在这里:
http://docs.confluent.io/current/clients/confluent-kafka-go/index.html#Consumer.CommitOffsets

除了API之外,kafka-consumer-groups命令还提供了移动消费者组位置的功能。这是在Apache Kafka 0.11中发布的。

英文:

You could use CommitOffsets and just commit back to the offset you want to rewind to. The next poll will start from that offset.

CommitOffsets is documented here:
http://docs.confluent.io/current/clients/confluent-kafka-go/index.html#Consumer.CommitOffsets

Outside of the API, there's functionality in the kafka-consumer-groups command to move the position of consumer groups as well. This is released with Apache Kafka 0.11.

huangapple
  • 本文由 发表于 2017年9月2日 08:37:51
  • 转载请务必保留本文链接:https://go.coder-hub.com/46009460.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定