英文:
How to manage data consistency with Chronicle queue
问题
以下是翻译好的部分：
我们一直在使用Apache Kafka构建一组微服务，作为消息总线来进行通信。一些微服务还与数据库或JMS消息代理（如IBMM MQ）互动，或两者兼有。
由于Kafka无法保证“仅一次”（exactly once），因此我们遇到了许多问题，需要付出很多努力来防止重复发生。除此之外，我们实现的性能远低于所宣传的性能。
由于低延迟是我们作为概念验证的主要要求之一，我们用Chronicle Queues替代了Kafka主题并进行了尝试。根据批处理大小以及我们在TPS（每秒事务数）方面对系统的压力程度，Chronicle版本的性能比Kafka版本快了至少10倍，平均快了至少10倍。
无论如何，低延迟只是要求之一。数据一致性至关重要。在任何情况下，我们都不希望重复消息或丢失消息。查阅Chronicle Queue文档以及在网上搜索并没有帮助我了解如何以事务方式使用Chronicle Queue。考虑与其他资源（JMS + DB）的集成时，可能更喜欢一种像XA事务的方式。
因此，我的问题是，在使用Chronicle Queue来确保数据一致性时，通常采用哪些常见模式？提供一篇好文章的链接、一些建议或一些代码示例，以指导我朝正确的方向前进，将足够满足我的需求。
提前感谢您。
英文:
We have been building a bunch of micro services using Apache Kafka as a messaging bus to communicate between them. Some micro services also interact with a database or a JMS message broker (IBMM MQ) or both.
We had lots of problems caused by the fact that Kafka does not guaranty exactly once resulting in lots of efforts put in place to prevent duplicates happening. In addition to this the performance we achieved is far for what is being advertised.
Because low latency is one of our main requirements as a Prof Of Concept we replaced Kafka topics with Chronicle Queues and gave it a go. Depending on the batch sizes and how hard we smashed our systems in terms of TPS (transactions per second) it came as a nice surprise that Chronicle version was anywhere between six times to two hundreds times faster than the Kafka version with an average of at least ten times faster.
Anyway low latency is only one of the requirements. Data consistency is paramount. Under no circumstances we would like to duplicate a message or loose one. Going through Chronicle Queue documentation as well as searching the web did not take me anywhere about how to use Chronicle Queue in a transactional way. Considering integration with some other resources (JMS + DB) some XA transaction like would be preferred.
So my question is what are the common patterns applied when using Chronicle Queue to assure data consistency? A link to some good article or some suggestions or a bit of code example to put me on the right direction would would suffice.
Thank you in advance.
答案1
得分: 0
如果您需要处理主机故障，您将需要我们的复制服务，它是Chronicle Queue Enterprise的一部分。
然而，如果您想处理进程故障，您可以使用命名的尾随器（Named Tailers）。这允许您控制并继续从它之前的索引位置进行尾随。
当您打开一个DocumentContext时，您可以控制它是否应该继续或不继续。
例如：
try (ExceptTailer tailer = queue.createTailer("name")) {
// 在循环中
try (DocumentContext dc = tailer.readingDocument()) {
if (!dc.isPresent()) {
// 执行其他操作
}
// 处理消息
} catch(Throwable t) {
dc.rollbackOnClose();
throw t;
}
在这个示例中，只有在正常完成时索引才会移动。如果不正常完成，您将不会在此尾随器上再次看到该消息（即使重新启动），除非您明确更改索引。
您还可以通过打开相同的尾随器以另一个进程的方式来监视索引。
这是我进行的一个将Kafka与Chronicle Queue进行比较的基准测试：https://chronicle.software/benchmarking-kafka-vs-chronicle-for-microservices-which-is-750-times-faster/
英文:
If you need to handle the failure of a host you will need our replication service which is part of Chronicle Queue Enterprise.
However, if you want to handle the failure of a process, you can use Named Tailers. This allows you to control and continue a tailer from the index it was up to.
When you open a DocumentContext you can control whether it should continue or not.
e.g.
try (ExceptTailer tailer = queue.createTailer("name")) {
// in a loop
try (DocumentContext dc = tailer.readingDocument()) {
if (!dc.isPresent()) {
// do something else
}
// process message
} catch(Throwable t) {
dc.rollbackOnClose();
throw t;
}
In this example, the index will only move on if it completes normally. If it doesn't complete normally, you won't see that message again on this tailer (event on restart) unless you explicitly change the index.
You can also monitor the index in another process this way by opening the same tailer.
Here is a benchmark I did comparing Kafka to Chronicle Queue https://chronicle.software/benchmarking-kafka-vs-chronicle-for-microservices-which-is-750-times-faster/
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论