增加 DynamoDB 中的分片数量以并行启动更多 lambda [英] Increase number of shards in DynamoDB to spin up more lambdas in parallel

问题描述

我目前正在使用 DynamoDB 流来处理使用 lambda 函数更改的集合值，但是，目前，我只并行运行两个 lambda 实例，这不足以处理所有传入的数据和 lambda 函数刚刚排队.

I'm currently using DynamoDB streams to process changed collection values with lambda functions, however, currently, I'm only running two lambda instances in parallel, which is not enough to process all the incoming data and lambda functions are just queued up.

从 aws 文档中，我可以看到可以并行运行的 lambda 数量与 DynamoDB 的分片数量成正比:

From aws documentation I can see that number of lambdas that can run in parallel is proportional to the number of shards of your DynamoDB:

如果您创建一个 Lambda 函数来处理来自基于流的服务(Amazon Kinesis Streams 或 DynamoDB 流)，每个流的分片数是并发单位.如果你的流有 100 个活动分片，将有 100 个 Lambda 函数同时运行.然后，每个 Lambda 函数在按照它们到达的顺序分片.

If you create a Lambda function that processes events from stream-based services (Amazon Kinesis Streams or DynamoDB streams), the number of shards per stream is the unit of concurrency. If your stream has 100 active shards, there will be 100 Lambda functions running concurrently. Then, each Lambda function processes events on a shard in the order that they arrive.

所以我的问题是，如何增加 DynamoDB 的分片数量?甚至可能吗?我在设置中找不到如何设置它.

So my question is, how do I increase the number of shards of my DynamoDB? Is it even possible? I couldn't find how to set it up in the settings.

推荐答案

不，不能手动控制 DDB UpdateStream 中的分片数量.DDB 通过创建尽可能多的分片以匹配传入的更新速率来自动为您处理.

No, its not possible to manually control number of shards in DDB UpdateStream. DDB automatically handles that for you by creating as many shards to match the incoming rate of updates.

理想情况下，发生在 DDB 表上的更新应该流经某个分片(发生在同一条记录上的更新将始终进入同一个分片，这意味着它们是根据您的 hashKey 进行分区的).您的更新流也是按时间顺序排列的，因此对同一记录的更新最终(或者说排队)在同一个分片中，以便最终处理器按它们发生的顺序处理它们.

Ideally updates happening to your DDB table is supposed to flow through some shard (updates happening to same record will always go to same shard meaning they are partitioned based on your hashKey). It is your stream of updates that too in chronological order thus updates over same record end up (or say queued up) in same shard so that end processor process them in sequence they happened.

每个分片对输入和输出数据都有自己的吞吐能力，除非需要更多分片来支持即将到来的表更新速率(在 DDB 更新流的情况下，表上的写入 tps 很高，目前无法处理的分片数)

Each shard has its own throughput capacity for in and out of data unless there is need of more shards to support in coming rate of updates on table (which in case of DDB updates streams is high write tps on your table, which current number of shards can't handle)

这篇关于增加 DynamoDB 中的分片数量以并行启动更多 lambda的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！