每个 kinesis 分片有多个消费者 [英] multiple consumers per kinesis shard

问题描述

我了解到每个 kinesis 流可以有多个消费者应用程序.

I read you can have multiple consumer apps per kinesis stream.

http://docs.aws.amazon.com/kinesis/latest/dev/developing-consumers-with-kcl.html

但是，我听说每个分片只能拥有一个消费者.这是真的?我没有找到任何文档来支持这一点，并且无法想象如果多个消费者从同一个流中读取会怎样.当然，这并不意味着生产者需要为不同的消费者在不同的分片中重复内容.

however, I heard you can only have on consumer per shard. Is this true? I don't find any documentation to support this, and can't imagine how that could be if multiple consumers are reading from the same stream. Certainly, it doesn't mean the producer needs to repeat content in different shards for different consumers.

推荐答案

Kinesis Client Library 在后台启动线程，每个线程监听流中的 1 个分片.您无法通过多个线程连接到分片，这是设计使然.

Kinesis Client Library starts threads in the background, each listens to 1 shard in the stream. You cannot connect to a shard over multiple threads, that is by-design.

http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-scaling.html

例如，如果您的应用程序在一个 EC2 实例上运行，并且正在处理一个具有四个分片的 Amazon Kinesis 流.这个实例有一个 KCL 工作者和四个记录处理器(一个记录每个分片的处理器).这四个记录处理器运行在在同一个进程中并行.

For example, if your application is running on one EC2 instance, and is processing one Amazon Kinesis stream that has four shards. This one instance has one KCL worker and four record processors (one record processor for every shard). These four record processors run in parallel within the same process.

在上面的解释中，术语KCL 工作器"是指 Kinesis 消费者应用程序.不是线程.

In the explanation above, the term "KCL worker" refers to a Kinesis consumer application. Not the threads.

但在下面，相同的KCL worker"术语指的是应用程序中的Worker"线程；这是一个可运行的.

But below, the same "KCL worker" term refers to a "Worker" thread in the application; which is a runnable.

通常，当您使用 KCL 时，您应该确保实例数不超过分片数量(故障备用目的除外).每个分片是恰好由一名 KCL 工人处理，并且恰好有一名相应的记录处理器，因此您永远不需要多个实例来处理一个分片.

Typically, when you use the KCL, you should ensure that the number of instances does not exceed the number of shards (except for failure standby purposes). Each shard is processed by exactly one KCL worker and has exactly one corresponding record processor, so you never need multiple instances to process one shard.

请参阅Worker.java KCL 源代码中的类.

See the Worker.java class in KCL source.

这篇关于每个 kinesis 分片有多个消费者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！