如何使用Lambda函数处理SQS队列(不通过计划的事件)? [英] How to process SQS queue with lambda function (not via scheduled events)?
问题描述
这是我正在尝试的简化方案:
http请求->(网关API + lambda A)-> SQS->(lambda B ?????)-> DynamoDB
因此它应该如下所示工作:来自许多http请求的数据(例如，每秒高达500个) 由我的lambda函数A放入SQS队列.然后另一个函数B处理该队列: 读取多达10个项目(在某些定期的基础上)，然后使用BatchWriteItem将其写入DynamoDB.
问题是我不知道如何触发第二个lambda函数.应该每秒多次(或至少每秒一次)频繁调用它，因为我需要从队列中获取所有数据以尽快进入DynamoDB(这就是为什么通过计划的事件调用lambda函数B的原因，如我什至宁愿丢失一些记录，而不是增加在DynamoDB中接收和存储记录之间的间隔
UPD::如果有人感兴趣，我已经找到了如何在节流的情况下使aws-sdk跳过自动重试:有一个特殊参数解决方案
[这不能直接回答您的明确问题，因此根据我的经验，它将被否决:)但是，我将回答您尝试解决的基本问题解决.]
我们采用大量传入请求并将其馈送到AWS Lambda函数以按节奏方式写入DynamoDB的方法，是用Amazon Kinesis流替换拟议架构中的SQS.
Kinesis流可以驱动AWS Lambda函数.
Kinesis流保证了对于任何给定密钥的已传递消息的顺序(对于有序的数据库操作而言很好).
Kinesis流让您指定可以并行运行多少个AWS Lambda函数(每个分区一个)，这些函数可以与DynamoDB的写入容量配合使用.
Kinesis流可以在一个AWS Lambda函数调用中传递多个可用消息，从而可以进行进一步的优化.
注意:实际上是从Amazon Kinesis流读取然后调用该函数的AWS Lambda服务，而不是Kinesis流直接调用AWS Lambda.但有时在Kinesis驱动时更容易可视化.给用户的结果几乎相同.
Here is the simplified scheme I am trying to make work:
http requests --> (Gateway API + lambda A) --> SQS --> (lambda B ?????) --> DynamoDB
So it should work as shown: data coming from many http requests (up to 500 per second, for example) is placed into SQS queue by my lambda function A. Then the other function, B, processes the queue: reads up to 10 items (on some periodical basis) and writes them to DynamoDB with BatchWriteItem.
The problem is that I can't figure out how to trigger the second lambda function. It should be called frequently, multiple times per second (or at least once per second), because I need all the data from the queue to get into DynamoDB ASAP (that's why calling lambda function B via scheduled events as described here is not a option)
Why don't I want to write directly into DynamoDB, without SQS?
That would be great for me to avoid using SQS at all. The problem that I am trying to address with SQS is DynamoDB throttling. Not even throttling itself but the way it is handled while writing data to DynamoDB with AWS SDK: when writing records one by one and getting them throttled, AWS SDK silently retries writing, resulting in increasing of the request processing time from the http client's point of view.
So I would like to temporarily store data in the queue, send response "200 OK" back to client, and then get queue processed by separate function, writing multiple records with one DynamoDB's BatchWriteItem call (which returns Unprocessed items instead of automatic retry in case of throttling). I would even prefer to lose some records instead of increasing the lag between a record being received and stored in DynamoDB
UPD: If anyone is interested, I have found how to make aws-sdk skip automatic retries in case of throttling: there is a special parameter maxRetries. Anyway, going to use Kinesis as suggested below
[This doesn't directly answer your explicit question, so in my experience it will be downvoted :) However, I will answer the fundamental problem you are trying to solve.]
The way we take a flood of incoming requests and feed them to AWS Lambda functions for writing in a paced manner to DynamoDB is to replace SQS in the proposed architecture with Amazon Kinesis streams.
Kinesis streams can drive AWS Lambda functions.
Kinesis streams guarantee ordering of the delivered messages for any given key (nice for ordered database operations).
Kinesis streams let you specify how many AWS Lambda functions can be run in parallel (one per partition), which can be coordinated with your DynamoDB write capacity.
Kinesis streams can pass multiple available messages in one AWS Lambda function invocation, allowing for further optimization.
Note: It's really the AWS Lambda service that reads from Amazon Kinesis streams then invokes the function, and not Kinesis streams directly invoking AWS Lambda; but sometimes it's easier to visualize as Kinesis driving it. The result to the user is nearly the same.
这篇关于如何使用Lambda函数处理SQS队列(不通过计划的事件)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
