限制从AWS SQS触发的AWS Lambda的并发调用(保留的并发性被忽略)? [英] Limit concurrent invocation of a AWS Lambda triggered from AWS SQS (Reserved concurrency ignored)?
问题描述
对我来说，这似乎是我开始时的一个简单用例，但事实却比我预期的要难得多.
To me this seemed like a simple use case when I started, but it turned out a lot harder than I had anticipated.
问题
我有一个AWS SQS充当触发工作人员AWS Lambda的作业队列.但是，由于工作Lambda共享不可扩展的资源，因此将并发运行的Lambda的数量限制为(例如)不超过5个同时运行的Lambda很重要.
I have an AWS SQS acting as a job queue that triggers a worker AWS Lambda. However since the worker lambdas are sharing non-scalable resources it is important to limit the number of concurrent running lambdas to (for the sake of example) no more than 5 lambdas running simultaneously.
保留的并发性还限制了最大并发性.功能，并应用于整个功能
Reserved concurrency also limits the maximum concurrency for the function, and applies to the function as a whole
但是，SQS似乎完全忽略了将 Reserved Concurrency -property设置为5，在本例中，队列 Messages in Flight -property显示接近20-30个并发执行，具体取决于放入队列中的消息量.
However, setting the Reserved concurrency-property to 5 seems to be completely ignored by SQS, with the queue Messages in Flight-property in my case showing closer to 20-30 concurrent executions depending on the amount of messages put into the queue.
问题
The closest I have come to a solution is to use a SQS FIFO queue and setting the MessageGroupId to a value of either randomly selecting or alternating between 1-5. However, due to uneven workload this is not optimal as it would be better to have the concurrency distributed by actual workload rather than by chance.
我还尝试将AWS Step Functions用作地图状态具有MaxConcurrency参数，该参数似乎在小型作业队列上运行良好，但是由于每个状态的输入/输出限制为32kb，因此在我的系统中这是不可行的用例.
I have also tried using the AWS Step Functions as the Map-state has a MaxConcurrency parameter, which seemed to work well on small job queues, but due to each state having an input/output limit of 32kb, this was not feasible in my use-case.
有人找到更好或替代的解决方案吗?还有其他应该使用保留并发的方式吗?
Has anyone found a better or alternative solution? Are there any other ways Reserved concurrency is supposed to be used?
类似
这里有一些类似的问题，但是我认为我的问题有所不同，因为我对限制触发的行为与SQS有所不同.
Here are some similar questions I have found, but I think my question is different because I am not interested in limiting the total number of invocation, and (although I have not tried it myself) I can not see why triggers from S3 or Kinesis Steam would behave different from SQS.
推荐答案
根据AWS文档，AWS SQS不考虑保留的并发性.如果要处理的批次数量大于保留的并发数量，则您的消息可能最终死信队列中:
According to AWS docs AWS SQS doesn't take into account reserved concurrency. If number of batches to be processed is greater than reserved concurrency, your messages might end up in a dead-letter queue:
如果函数返回错误，或者由于位于最大并发性，附加处理可能会成功尝试.使邮件有更好的机会在处理之前将其发送到死信队列，然后在源队列的重新驱动策略至少为5. https://docs.aws.amazon.com/lambda/Latest/dg/with-sqs.html
您可以查看本文以了解详细信息: https://zaccharles.medium.com/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0
You can check this article for details: https://zaccharles.medium.com/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0
这篇关于限制从AWS SQS触发的AWS Lambda的并发调用(保留的并发性被忽略)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
