英文:
How to balance kinesis shards across several record processor?
问题
我目前正在使用Golang版本编写简单的Kinesis Client Library（KCL）。我希望我的简单KCL具备的一个功能是在多个记录处理器和EC2实例之间进行负载均衡。例如，我有两个记录处理器（将在单独的EC2实例中运行）和四个Kinesis分片。负载均衡功能将允许每个记录处理器处理两个Kinesis分片。
我了解到Java KCL已经实现了这个功能，但我在库中找不到实现。我的问题是，我该如何在Golang中实现这个功能？谢谢。
英文:
I am currently writing the simple Kinesis Client Library (KCL) in Golang version. One of the features that I want it for my simple KCL is load balancing shards across multiple record processors and EC2 instances. For example, I have two record processors (which will run in the separate EC2 instance) and four Kinesis shards. The load balancing feature will allow each record processors to process two Kinesis shards.
I read that Java KCL implemented this but I can't find the implementation in the library. My question is how am I going to implement this feature in Golang? Thank you.
答案1
得分: 9
KCL已经为您执行负载均衡。
以下是它的基本工作原理的简要描述（请记住，这只是基础知识，随着亚马逊改进逻辑，可能会有所变化）：
- 当一个工作节点（可能处理多个分片）启动时，它会检查一个中央的DynamoDB数据库，以确定哪些分片由工作节点拥有（如果需要，会创建该数据库）。这个数据库被称为“租赁”表。
- “租赁”是工作节点和分片之间的关系。
- 工作节点将处理它拥有有效租赁的分片的记录。
- 如果工作节点在租赁到期之前没有发出“心跳”（通常每几秒钟发出一次心跳），租赁将过期。这个“心跳”实际上是更新DynamoDB记录。
- 它检查Kinesis流中哪些分片是可用的，并在需要时更新表格。
- 如果有任何租赁已过期，工作节点将尝试获取该租赁的所有权 - 在数据库级别上，使用shardId作为键，并将其workerId写入其中。
- 如果一个工作节点启动时，所有分片都已被占用，它将检查“平衡”情况 - 如果它检测到不平衡（例如：“我没有拥有任何分片，而其他工作节点拥有10个分片”），它将启动“窃取分片”协议 - 旧的工作节点停止处理该分片，新的工作节点开始处理。
当然，您可以自由地查看KCL在GitHub上的源代码：https://github.com/awslabs/amazon-kinesis-client - 希望这个解释能让您更好地理解KCL并根据您的需求进行调整。
英文:
KCL already does load balancing for you.
Here's a basic description of how it works today (keep in mind this is just the basics and is subject to change as Amazon improves the logic):
- When a worker (which may process multiple shards) starts up, it checks a central DynamoDB database for which shards are owned by workers (creating that database if necessary). This is the "leasing" table.
- A "lease" is a relationship between a worker and a shard
- Workers will process records for shards it owns an unexpired lease for
- Leases expire if the worker hasn't emitted a "heartbeat" for the lease before it expires (typically every few seconds) - this heartbeat essentially updates the DDB record
- It checks Kinesis stream for which shards are available, and updates the table if needed
- If any leases are expired, the worker will try to take ownership of the lease - at database level, use shardId as key and write it's workerId there.
- If a worker starts and all shards are already taken, it checks to see what the "balance" is - if it detects an imbalance (ie: "I own 0 shards and some other worker owns 10 shards"), it initiates a "steal shard" protocol - the old worker stops processing for that shard, and the new worker starts
You're of course free to examine the source code for KCL on github: https://github.com/awslabs/amazon-kinesis-client - hopefully this explanation gives you more context for how to understand KCL and adapt it to your needs.
答案2
得分: 0
在你开始编写自己的客户端之前...看起来已经有一些人已经做过了：
你还有另一个选择，那就是使用KCL MultiLangDaemon。你可以安装一个小的运行程序来处理所有的负载均衡，然后只需监听守护进程发送的消息并将其提交回去。
https://github.com/awslabs/amazon-kinesis-client#amazon-kcl-support-for-other-languages
英文:
Before you start writing your own client...it looks like there are some people who have already done this:
Another option you have is the KCL MultiLangDaemon. You can install a small runner program that does all the balancing for you, and then you just listen to the messages the daemon sends you and commit them back.
https://github.com/awslabs/amazon-kinesis-client#amazon-kcl-support-for-other-languages
答案3
得分: 0
对于对这个主题感兴趣的人，VMWare编写了Java lib的Go实现：https://github.com/vmware/vmware-go-kcl/
不过，截至目前，它不支持租约窃取：https://github.com/vmware/vmware-go-kcl/issues/4
英文:
For people interested in the topic, VMWare wrote the Go implementation of the Java lib: https://github.com/vmware/vmware-go-kcl/
At the time of writing, though, it doesn't support lease stealing: https://github.com/vmware/vmware-go-kcl/issues/4
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论