英文:
C++ client : Ensuring Message Consumption Consistency in Aeron with Multiple Consumers on same channel
问题
I have a scenario where I'm using Aeron messaging library with multiple consumers consuming from the same channel. I want to ensure that each message is consumed by only one consumer to avoid duplication and guarantee consistency across consumers. In Kafka, we have the concept of committing offsets to track the progress of each consumer. Is there a similar mechanism or approach in Aeron to achieve message consumption consistency?
在我的情况下,我正在使用 Aeron 消息库,多个消费者从同一通道消费消息。我想确保每条消息只被一个消费者消费,以避免重复,并确保在消费者之间保持一致性。在 Kafka 中,我们有提交偏移量的概念,用于跟踪每个消费者的进度。Aeron 中是否有类似的机制或方法来实现消息消费的一致性?
In my setup, I have multiple consumers that subscribe to the same Aeron channel and I want to make sure that messages are distributed across consumers in a balanced manner. I'm aware that Aeron provides a broadcast approach by default, where each consumer receives a copy of every message. However, I'm interested in a scenario where each message is consumed by only one consumer.
在我的设置中,我有多个消费者订阅同一 Aeron 通道,我想确保消息以均衡的方式分布给各个消费者。我知道 Aeron 默认提供广播方式,其中每个消费者都会收到每条消息的副本。但是,我对一种情况感兴趣,即每条消息只由一个消费者消费。
I have explored the Aeron documentation and API, but I couldn't find a built-in mechanism similar to Kafka's offset committing. I would like to know if there are any recommended patterns or techniques to achieve message consumption consistency in Aeron with multiple consumers.
我已经研究了 Aeron 的文档和 API,但是我没有找到类似于 Kafka 提交偏移量的内置机制。我想知道是否有任何推荐的模式或技术,可以在具有多个消费者的情况下实现 Aeron 中的消息消费一致性。
My specific requirements are as follows:
我的具体要求如下:
Guarantee that each message is consumed by only one consumer to avoid duplication.
确保每条消息只由一个消费者消费,以避免重复。
Ensure consistency across consumers, so that no two consumers process the same message in the same channel.
确保各个消费者之间的一致性,以确保在同一通道中没有两个消费者处理相同的消息。
Ideally, have a mechanism to track the progress of each consumer, similar to Kafka's offset committing.
理想情况下,希望有一种机制来跟踪每个消费者的进度,类似于 Kafka 的提交偏移量。
I would appreciate any insights, recommendations, or best practices for handling this scenario in Aeron. If there are any C++ code examples or configuration suggestions, they would be highly valuable.
我会感激任何有关在 Aeron 中处理这种情况的见解、建议或最佳实践。如果有任何 C++ 代码示例或配置建议,那将非常有价值。
Thank you in advance for your help!
提前感谢您的帮助!
英文:
I have a scenario where I'm using Aeron messaging library with multiple consumers consuming from the same channel. I want to ensure that each message is consumed by only one consumer to avoid duplication and guarantee consistency across consumers. In Kafka, we have the concept of committing offsets to track the progress of each consumer. Is there a similar mechanism or approach in Aeron to achieve message consumption consistency?
In my setup, I have multiple consumers that subscribe to the same Aeron channel and I want to make sure that messages are distributed across consumers in a balanced manner. I'm aware that Aeron provides a broadcast approach by default, where each consumer receives a copy of every message. However, I'm interested in a scenario where each message is consumed by only one consumer.
I have explored the Aeron documentation and API, but I couldn't find a built-in mechanism similar to Kafka's offset committing. I would like to know if there are any recommended patterns or techniques to achieve message consumption consistency in Aeron with multiple consumers.
My specific requirements are as follows:
Guarantee that each message is consumed by only one consumer to avoid duplication.
Ensure consistency across consumers, so that no two consumers process the same message in same channel.
Ideally, have a mechanism to track the progress of each consumer, similar to Kafka's offset committing.
I would appreciate any insights, recommendations, or best practices for handling this scenario in Aeron. If there are any C++ code examples or configuration suggestions, they would be highly valuable.
Thank you in advance for your help!
答案1
得分: 2
TL;DR: 此模型不受 Aeron 支持,您不太可能找到类似操作的示例。
将 Aeron 强制以这种方式工作将会很困难。例如,您如何协调多个订阅在不同机器上订阅同一组播 UDP 通道?如何确保每条消息都被实际处理,即如果其中一个消费者在处理消息的中途崩溃会怎么样?需要基础设施来提供这些严格的保证非常困难,因此许多系统倾向于允许冗余、幂等操作和至少一次交付。操作的冗余提供更好的可靠性,而更简单的实现模型通常会表现得更好*。
*更严格的保证,如避免重复,将需要更多组件之间的通信,以确保满足一致性要求。这会导致更高的延迟和较低的吞吐量。
英文:
TL;DR: This model is not supported by Aeron and you are unlikely to find examples to do something similar.
Forcing Aeron to work this way would prove difficult. E.g. how would you co-ordinate multiple subscriptions to the same multicast UDP channel across separate machines? How do you ensure that each message is actually processed, i.e. what if one of the consumers crashes half way through processing the message? Requiring an infrastructure to provide these strict guarantees is very difficult so many systems tend toward allowing redundant, idempotent operations and at least once delivery. The redundancy of operations provides better reliability and the simpler implementation model will tend to perform better*.
*Stricter guarantees like avoiding duplication will need more communication between to components to ensure that the consistency requirements are met. This leads to higher latency and lower throughput.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论