英文:
Can I add arbitrary key to all messages created by Kafka Connect from a csv file?
问题
我需要将由Kafka Connect（spooldir）从CSV文件创建的所有消息标记为来自同一文件。我可以为这些消息添加任意键吗？（例如键可以是文件的名称）
此外，如果未处理目录中有两个文件，Kafka Connect会逐个处理它们吗？不会将来自不同文件的消息混合在一个主题流中吗？
英文:
I need to mark all messages created by Kafka Connect (spooldir) from a csv file as coming from one same file. Can I add arbitrary key to these messages? (for example the key being the name of the file)
Also if there are two files in unprocessed directory, Kafka Connect would process them one by one? Not mixing messages from different files in a topic stream?
答案1
得分: 1
你可以使用 InsertField$Key 转换来实现这个目的。
https://docs.confluent.io/platform/current/connect/transforms/insertfield.html
然而，这会添加静态数据，因此无法直接访问已读取的文件。换句话说，来自不同文件的事件最终会插入相同的键。
> 不要在主题流中混合来自不同文件的消息
所有文件的所有行将被读取到一个接收主题中；因此，它们将在分区内部和跨分区之间混合在一起。
英文:
You can use InsertField$Key transform for this.
https://docs.confluent.io/platform/current/connect/transforms/insertfield.html
However, this adds static data, and therefore will not have direct access to the file that had been read. In other words, events from differing files will end up having the same key being inserted.
> Not mixing messages from different files in a topic stream
All lines of all files will be read into one sink topic; therefore will be mixed, within and across partitions.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论