使用Amazon KCL的检查点记录会引发ProvisionedThroughputExceededException [英] Checkpointing records with Amazon KCL throws ProvisionedThroughputExceededException

问题描述

在同时检查许多事件时，我们正在遇到 ProvisionedThroughputExceededException .

We are experiencing a ProvisionedThroughputExceededException upon checkpointing many events together.

异常堆栈跟踪如下:

    com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for shard shardId-000000000000 in stream mystream under account accountid. (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: ea36760b-9db3-0acc-bbe9-87939e3270aa)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1645)
at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1621)
at com.amazonaws.services.kinesis.AmazonKinesisClient.getShardIterator(AmazonKinesisClient.java:909)
at com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy.getIterator(KinesisProxy.java:291)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator.validateSequenceNumber(SequenceNumberValidator.java:79)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.RecordProcessorCheckpointer.checkpoint(RecordProcessorCheckpointer.java:120)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.RecordProcessorCheckpointer.checkpoint(RecordProcessorCheckpointer.java:90)

如您在此处看到的，在以下位置引发了异常

As you can see here, the exception is raised at

RecordProcessorCheckpointer.java:90

在KCL库中.检查点与超出吞吐量有什么关系?

inside the KCL library. What does checkpointing has to do with exceeding the throughput?

推荐答案

运动是受速率限制的，

PutRecord请求最多只能处理所涉及的分片上所设置的吞吐量的限制.超出此范围将抛出 ProvisionedThroughputExceededException

PutRecord requests can only process up to the limit of the provisioned throughput on the involved shard. exceeding this will throw ProvisionedThroughputExceededException

一种明显的解决方案是将流的碎片分成两部分，然后将哈希键空间均匀划分.如果您的指标在单个分片的范围内，这似乎是不必要的，但是可以说，如果您在开始的 500ms 中使用了 1000个事务/秒的写限制，则您的活动该分片将在剩余的一半内被节流，因此您无法避免使用单个分片进行节流.

Obvious solution would be splitting stream's shard into two and divide the hash key space evenly. It might look unnecessary if your metrics are within the limits of a single shard but lets say if you use your limit of 1000 transactions/sec write capacity in first 500ms your activity for that shard will be throttled for the remaining half so there is no way you can avoid throttling with a single shard.

您可以为受限制的请求配置短暂延迟后的自动重试.如果有任何示例，请查看SDK的文档.

You can configure automatic retries after short delays for your throttled requests. check your SDK's documentation if there is any examples of this.

这篇关于使用Amazon KCL的检查点记录会引发ProvisionedThroughputExceededException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！