英文:
Is there any standard solution for collecting versioned messages, and forwarding only the message with the highest version?
问题
让我们假设我有一个消息生产者,它在很短的时间内发送大量带有版本 V1、V2、...、VN 的消息。
我们有一个服务,不需要对每条这样的消息做出反应。我们可以采用懒惰的方法,在一分钟内收集消息,然后将带有最新版本的消息发送出去。
想了解是否有标准的解决方法?
请注意,我正在使用AWS,所以他们提供的任何适用于这种情况的服务也可以。
英文:
Let's say I have a message producer, which sends a very high number of messages with versions V1, V2, ..., VN, in a short timespan.
We have a service, which is not required to react to every such message. We can have a lazy approach, where we collect messages for a minute, and send forward the message with the latest version.
Wanted to understand whether there is a standard solve for this?
Note that I am working on AWS, so any service that they provide for this use case would also work out fine.
答案1
得分: 1
不要回答我要翻译的问题。以下是要翻译的内容:
"Instead of using a Queue (which requires messages to be specifically read), you could use a Stream."
"From What Is Amazon Kinesis Data Streams?:
>You can use Amazon Kinesis Data Streams to collect and process large streams of data records in real time. You can create data-processing applications, known as Kinesis Data Streams applications. A typical Kinesis Data Streams application reads data from a data stream as data records."
"You could write some code that reads the stream for a given time period and determines which records have the highest version number."
"To explain the different between a Queue and a Stream, think of somebody (the 'Producer') writing information on a piece of paper and throwing it into a jar. It keeps doing this, and the jar gets lots of pieces of paper. The jar is the Queue and the pieces of paper are the Messages. When somebody (the 'Consumer') wants to process a message, they reach into the jar and retrieve a random message. If you are using a First-In-First-Out (FIFO) queue, then replace the jar with a shoebox and always add the pieces of paper from one end and always retrieve them from the other end."
"In contrast, a Stream is like a film strip where the film frames (Messages) are always kept in the correct order. Consumers can go back to an earlier 'frame' of film and start reading the stream. Multiple consumers can read the stream, each starting at the same or different frames of the the film. They can even go backwards and re-read some frames. This differs from queues where 'consuming' a message from the queue also removes it from the queue."
"For your situation, you could send the messages to a Kinesis Data Stream instead of an Amazon SQS queue. Your app could then look at a segment of the stream, grab all the messages, figure out which ones have the highest version and then just forward those messages. It is a more complex program than one that merely pulls a message off a queue, but it gives you the benefit of being able to look at multiple messages simultaneously (and even repeatedly) to determine which one to forward."
"If you only want to send the 'very latest' message, then it is even easier -- just grab the last message from the stream and forward that one. No need to even look at earlier messages."
英文:
Instead of using a Queue (which requires messages to be specifically read), you could use a Stream.
From What Is Amazon Kinesis Data Streams?:
>You can use Amazon Kinesis Data Streams to collect and process large streams of data records in real time. You can create data-processing applications, known as Kinesis Data Streams applications. A typical Kinesis Data Streams application reads data from a data stream as data records.
You could write some code that reads the stream for a given time period and determines which records have the highest version number.
To explain the different between a Queue and a Stream, think of somebody (the 'Producer') writing information on a piece of paper and throwing it into a jar. It keeps doing this, and the jar gets lots of pieces of paper. The jar is the Queue and the pieces of paper are the Messages. When somebody (the 'Consumer') wants to process a message, they reach into the jar and retrieve a random message. If you are using a First-In-First-Out (FIFO) queue, then replace the jar with a shoebox and always add the pieces of paper from one end and always retrieve them from the other end.
In contrast, a Stream is like a film strip where the film frames (Messages) are always kept in the correct order. Consumers can go back to an earlier 'frame' of film and start reading the stream. Multiple consumers can read the stream, each starting at the same or different frames of the the film. They can even go backwards and re-read some frames. This differs from queues where 'consuming' a message from the queue also removes it from the queue.
For your situation, you could send the messages to a Kinesis Data Stream instead of an Amazon SQS queue. Your app could then look at a segment of the stream, grab all the messages, figure out which ones have the highest version and then just forward those messages. It is a more complex program than one that merely pulls a message off a queue, but it gives you the benefit of being able to look at multiple messages simultaneously (and even repeatedly) to determine which one to forward.
If you only want to send the 'very latest' message, then it is even easier -- just grab the last message from the stream and forward that one. No need to even look at earlier messages.
答案2
得分: 0
你可以使用SQS批处理来处理这些消息,并使用lambda/ecs/ec2来处理并将具有最高版本的消息转发到下游。
你可以配置lambda在一定时间内从队列中读取消息。
在这里:
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
为了避免以少量记录触发函数,你可以告诉事件源在最多5分钟的时间内缓冲记录,以配置批处理窗口。
英文:
You can you SQS Batch processing and process these messages using lambda/ecs/ec2 and forward the message with highest version downstream.
You can configure lambda to read from queue for a certain period of time
here:
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
To avoid invoking the function with a small number of records,
you can tell the event source to buffer records for up to 5 minutes by
configuring a batch window
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论