如何在多个记录处理器之间平衡 kinesis 分片? [英] How to balance kinesis shards across several record processor?

问题描述

我目前正在编写 Golang 版本的简单 Kinesis 客户端库 (KCL).我希望它用于我的简单 KCL 的功能之一是跨多个记录处理器和 EC2 实例的负载平衡分片.例如，我有两个记录处理器(将在单独的 EC2 实例中运行)和四个 Kinesis 分片.负载平衡功能将允许每个记录处理器处理两个 Kinesis 分片.

I am currently writing the simple Kinesis Client Library (KCL) in Golang version. One of the features that I want it for my simple KCL is load balancing shards across multiple record processors and EC2 instances. For example, I have two record processors (which will run in the separate EC2 instance) and four Kinesis shards. The load balancing feature will allow each record processors to process two Kinesis shards.

我读到 Java KCL 实现了这个，但我在库中找不到实现.我的问题是我将如何在 Golang 中实现这个功能?谢谢你.

I read that Java KCL implemented this but I can't find the implementation in the library. My question is how am I going to implement this feature in Golang? Thank you.

推荐答案

KCL 已经为您做了负载均衡.

KCL already does load balancing for you.

以下是它今天工作原理的基本描述(请记住，这只是基础知识，随着亚马逊改进逻辑可能会发生变化):

Here's a basic description of how it works today (keep in mind this is just the basics and is subject to change as Amazon improves the logic):

• 当工作器(可能处理多个分片)启动时，它会检查中央 DynamoDB 数据库，了解哪些分片归工作器所有(如有必要，创建该数据库).这是租赁"表.
  • 租约"是工人和分片之间的关系
  • 工作人员将处理其拥有未到期租约的分片的记录
  • 如果工作人员在租约到期之前(通常每隔几秒)没有为租约发出心跳"，则租约到期 - 此心跳实质上更新了 DDB 记录

  您当然可以在 github 上自由检查 KCL 的源代码:https://github.com/awslabs/amazon-kinesis-client - 希望这个解释为您提供更多关于如何理解 KCL 并使其适应您的需求的背景.

  You're of course free to examine the source code for KCL on github: https://github.com/awslabs/amazon-kinesis-client - hopefully this explanation gives you more context for how to understand KCL and adapt it to your needs.

  这篇关于如何在多个记录处理器之间平衡 kinesis 分片?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！