英文:
How to evolve local keyvaluestore schema
问题
我们有一个使用Kafka Streams处理数据的应用程序，我们使用Confluent Registry并以Avro格式定义模式。我们有一个名为paymentdetailstore的本地键值状态，键为payementDTO，其中包含付款ID和付款来源，值为paymentValueDTO，包含付款详细信息。现在我们想在键中引入一个新字段，比如originator，它可以为null。如果我们进行此更改，将生成一个新的模式ID。由于模式ID在序列化过程中嵌入在键中，我们将无法获取正确的值，使得我们现有的数据无法使用。是否有一种方法可以解决这个问题，而不使用名称中的版本信息？
英文:
We have an application that uses Kafka streams to process data, we are using confluent registry and defining schemas in Avro format. We have a local key-value state named as paymentdetailstore say with the key as payementDTO which consists of payment id and payment source and the value is paymentValueDTO with payment details. now we want to introduce a new field in key, say orignator which can be null. If we do this change it will generate a new schema-id. now since schema-id is embedded in key during serialization and we will not be able to fetch the correct value making our existing data unusable. Is there any way to counter it without using version information in name.
答案1
得分: 1
AvroSerde类来自Confluent已经维护了自己的ID缓存，没有理由自己这样做。
如果您构建了一个SpecificRecord serde类，并使用“版本2”运行您的代码，那么主题中的任何“版本1”记录将自动演变。
请记住，如果您更改了键，那么您也正在更改生产者的分区方案，这会导致KStreams中的排序问题。
英文:
The AvroSerde class from Confluent already maintains its own ID cache; there's no reason to do that on your own.
If you build a SpecificRecord serde class and run your code with "version 2", then any "version 1" record in the topic will automatically be evolved.
Keep in mind that if you change the key, you're also changing partitioning scheme of the producer, which causes ordering issues in KStreams
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论