英文:
Upgrading to Debezium Postgres Connector v2.0.1 causing schema incompatibility issues
问题
我有一个使用Kafka、Postgres的Debezium Connector和Schema注册表的应用程序。目前我正在运行Debezium Connector的1.9.3版本，一切正常运行。
最近，当我尝试将连接器版本升级到2.0.1时，我遇到了一些模式不兼容的问题。以下是错误消息 -
io.confluent.rest.exceptions.RestException: Schema being registered is incompatible with an earlier schema for subject
我查看了Debezium Connector的发布说明，强烈认为这可能是因为这个破坏性变更。
所有由Debezium使用的模式现在都在一个中心点中定义，具有适当的名称和版本（DBZ-4365，DBZ-5044）。如果使用模式注册表，这可能导致模式兼容性问题。
此外，根据我的探索，似乎使用Debezium事件展平的连接器不会遇到此问题。
有人能指导我解决这个问题吗？
英文:
I have an application that uses Kafka, Debezium Connector for Postgres, and a Schema registry. Currently I am running version 1.9.3 of Debezium Connector and everything runs fine.
Lately, when I tried upgrading the connector version to 2.0.1, I am facing some Schema incompatibility issues. Following is the error message -
io.confluent.rest.exceptions.RestException: Schema being registered is incompatible with an earlier schema for subject
I went through the release notes for Debezium Connector and strongly believes that this could be because of this breaking change.
All schemas used by Debezium are now defined in a central point, properly named and versioned (DBZ-4365, DBZ-5044). This can lead to schema compatibility issues if a schema registry is used.
Additionally, as per my explorations, it seems that connectors with Debezium event flattening do not face this issue.
Can anyone guide me to a solution for this?
答案1
得分: 0
I couldn't get a proper answer to this anywhere. Posting the solution that I finally ended up with.
So it seems like there is no officially supported way of fixing this. I reached out to the Debezium community and two possible solutions that they proposed were -
- Degrade the compatibility guarantees of schema registry to none and upgrade it to backward later sometime when all the schemas are upgraded to the latest version.
- Remove all the old schemas from the schema registry. Thus eliminating the compatibility issue.
Implementing any of the two solutions was not possible in my case!
For the first suggestion, it is hard to say when we will be able to upgrade the compatibility guarantees. It can only be done when all the connectors have processed at least one message after the upgrade. This is something that was very hard to assess in my case.
For the second suggestion, removing the schema from the schema registry would cause the readers to fail to read existing messages! This is because readers query the schema registry to get the schema.
I finally took the following approach:
- Paused all the connectors after the upgrade.
- Removed all the old schemas.
- Reset the offsets of all the schemas. Thus forcing them to re-process all the messages.
- Resumed the connectors.
The above process ensured that all the old schemas are replaced with the newer schema.
Although there is one catch, in my case, all the downstream consumers were idempotent (safe around handling duplicate messages). This is why this approach works. If your downstream consumers are not idempotent, this approach can get your system in an awful state.
英文:
I couldn't get a proper answer to this anywhere. Posting the solution that I finally ended up with.
So it seems like there is no officially supported way of fixing this. I reached out to the Debezium community and two possible solutions that they proposed were -
- Degrade the compatibility guarantees of schema registry to none and upgrade it to backward later sometime when all the schemas are upgraded to latest version.
- Remove all the old schemas from the schema registry. Thus eliminating the compatibility issue.
Implementing any of the two solutions was not possible in my case!
For the first suggestion, it is hard to say when we will be able to upgrade the compatibility guarantees. It can only be done when all the connectors have processed at least one message after the upgrade. This is something that was very hard to assess in my case.
For the second suggestion, removing the schema from the schema registry would cause the readers to fail to read existing messages! This is because readers query the schema registry to get the schema.
I finally took the following approach:
- Paused all the connectors after the upgrade.
- Removed all the old schemas.
- Reset the offsets of all the schemas. Thus forcing them to re-process all the messages.
- Resumed the connectors.
The above process ensured that all the old schemas are replaced with the newer schema.
Although there is one catch, in my case all the downstream consumers were idempotent (safe around handling duplicate messages). This is why this approach works. If your downstream consumers are not idempotent, this approach can get your system in an awful state.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论