Prevent duplicate message publish in Kafka topic if message is present in topic

I am working on one software which is middle system and works to transfer data from one to another system.

We are using Kafka with spring boot in it. I am recently started working this system and having limited knowledge on Kafka.

I am searching for solution on preventing duplicate message publishing to Kafka topic if same message is already present in topic and is not yet consumed.

For example -
Consider message "ABC" we have published to Kafka topic and is not yet consumed by consumer and we again received same message "ABC" then we will just skip this same message publishing.
Message "ABC" will be published again only if it is not present currently in Kafka topic.

Note - We do not have control on another system which send messages to us therefore there are situation where we received duplicate messages.

Thanks in advance for any ideas/solutions and suggestion for achieve the same.

Kafka has no control over this. It sees byte arrays and will not deserialize data to compare anything; every record produced is at a unique offset and there's no API available in the broker to know if "ABC" exists without consuming the entire topic (which could easily be TB of data) and is always going to be a linear scan for every new event.

Therefore, you'll need some other system with fast constant-time key/property lookups like Redis or an indexed MongoDB that tells you if that value has been seen and/or processed before.

Or, you write your consumer processing logic in an idempotent way, where duplicates don't affect anything. For example, a database update always overrides the latest seen information, even for the same ID or if the entire payload matches.