Kinesis最大分片读取数/秒和多个使用者 [英] Kinesis max shard reads/sec and multiple consumers

问题描述

因此，我有一个AWS Kinesis流，可以在其中为多个使用者发布事件.对于他们中的大多数来说，接收热数据很重要-这意味着其中许多人可能会同时轮询和读取最新数据.根据AWS文档，增加分片数量将提高并行度，而每个分片每秒读取数最大为5/sec.我的问题是是否(以及如何?)添加更多的分片是否可以帮助我的所有使用者都处于最新状态并尝试从同一分片读取新的传入数据的情况?似乎每秒读取数"限制会自动对您可以拥有的使用者数量(至少在需要始终更新的情况下)施加限制，或者我错过了什么?

So I have a AWS Kinesis stream where I publish events for multiple consumers. It is important for most of them to receive hot data - which means that many of them will possibly poll and read the latest data at the same time. According to the AWS documentation increasing the number of shards will increase the level of parallelism while the number of reads/sec can be max 5/sec per shard. My question is whether (and how?) would adding more shards help the situation where all my consumers are up-to-date and attempt to read new incoming data from the same shard? It seems to be that this reads per sec limitation automatically introduces a limitation on the number of consumers you can have (at least when they need to be updated at all times), or am I missing something?

推荐答案

是的，您是正确的.

在消费者中，我假设您将使用Amazon Kinesis Client(或KCL:​​ amazon- kinesis-client )作为API助手；并请看是否有一个参数"

In the consumers, I assume you'll use Amazon Kinesis Client (or KCL: amazon-kinesis-client) as API helper; and please take a look that there is a parameter "idleTimeBetweenReadsInMillis" in the consumer logic. That defines how much your application will poll the stream (the lower this value, more frequent your apps will poll).

无论您的流包含1个分片还是100个分片，您制作的分片都不能超过5个."

Whether your stream contains 1 shard or 100 shards, you cannot make more than 5 "GetRecords" requests per second for each shard. That is;

• 如果您有1个应用程序，则理论上最多可以使轮询间隔为200毫秒.
• 如果您有2个应用程序，则至少应为400毫秒.
• 如果您有3个应用程序，则至少应为600毫秒.
• 或者在您的3个应用程序中，其中两个可以以1000毫秒的速度进行轮询，最后一个可以以333ms的速度进行轮询.

您还可以为自己创建一个Kafka集群并对其性能进行基准测试. Kafka可能会提供更高的吞吐量.

You can also create a Kafka cluster for yourself and benchmark their performance. Kafka may give higher throughput.

请参阅以下答案，以比较Kafka和Kinesis概念: Kafka像Kinesis Stream上的偏移量一样?

See this answer for sample comparison between Kafka and Kinesis concepts: Kafka like offset on Kinesis Stream?

这篇关于Kinesis最大分片读取数/秒和多个使用者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！