每个运动碎片有多个消费者 [英] multiple consumers per kinesis shard

问题描述

我读到您可以在每个运动学流中拥有多个消费者应用程序.

I read you can have multiple consumer apps per kinesis stream.

http://docs.aws. amazon.com/kinesis/latest/dev/developing-consumers-with-kcl.html

但是，我听说您只能在每个分片上使用消费者.这是真的?我找不到任何文档来支持这一点，也无法想象如果多个使用者正在从同一流中读取内容，那会是怎样的情况.当然，这并不意味着生产者需要为不同的消费者重复使用不同分片中的内容.

however, I heard you can only have on consumer per shard. Is this true? I don't find any documentation to support this, and can't imagine how that could be if multiple consumers are reading from the same stream. Certainly, it doesn't mean the producer needs to repeat content in different shards for different consumers.

推荐答案

Kinesis Client Library在后台启动线程，每个线程侦听流中的1个分片.您不能通过多线程连接到分片，这是设计使然.

Kinesis Client Library starts threads in the background, each listens to 1 shard in the stream. You cannot connect to a shard over multiple threads, that is by-design.

http://docs.aws. amazon.com/kinesis/latest/dev/kinesis-record-processor-scaling.html

例如，如果您的应用程序在一个EC2实例上运行，并且 正在处理一个具有四个分片的Amazon Kinesis流.这个 实例有一个KCL工作程序和四个记录处理器(一个记录 每个分片的处理器).这四个记录处理器运行在 在同一过程中并行进行.

For example, if your application is running on one EC2 instance, and is processing one Amazon Kinesis stream that has four shards. This one instance has one KCL worker and four record processors (one record processor for every shard). These four record processors run in parallel within the same process.

在上面的解释中，术语"KCL工作者"是指Kinesis消费者应用程序.不是线程.

In the explanation above, the term "KCL worker" refers to a Kinesis consumer application. Not the threads.

但是在下面，相同的"KCL worker"一词指的是应用程序中的"Worker"线程；这是可运行的.

But below, the same "KCL worker" term refers to a "Worker" thread in the application; which is a runnable.

通常，当您使用KCL时， 您应确保实例数量不超过 分片数量(故障备用目的除外).每个分片都是 仅由一名KCL工人处理，并有一名对应的KCL工人 记录处理器，因此您不需要多个实例来处理一个 分片.

Typically, when you use the KCL, you should ensure that the number of instances does not exceed the number of shards (except for failure standby purposes). Each shard is processed by exactly one KCL worker and has exactly one corresponding record processor, so you never need multiple instances to process one shard.

请参见 查看全文