递归AWS Lambda函数调用-最佳实践 [英] Recursive AWS Lambda function calls - Best Practice

问题描述

我的任务是看一下基于AWS Lambda构建的服务，该服务执行长时间运行的打开和关闭VM的任务.请注意，我来自Azure团队，所以我不熟悉AWS服务的样式或最佳实践.

原始开发人员采用的方法是将整个工作负载发送给一个Lambda函数，然后让该函数承担一部分工作负载，然后递归调用剩余的工作负载，直到所有项目都消失(工作负载= 0)

伪代码:

 //假设这已作为一个整体发送到HTTP Lambda端点让工作量= [1、2、3、4、5、6、7、8]//Lambda HTTP端点函数Lambda(工作量){如果(！workload.length){返回不再工作！"}const toDo = load.splice(0，2)//获取前两个项目doWork(toDo)//然后...除了它用aws sdk构建一个新的HTTP请求Lambda(工作量)//3、4、5、6、7、8等} 

这似乎效率很低而且不可靠(如果我错了，请纠正我).我认为在此过程中存储了很多状态，这会导致很多故障点.

我的计划是建议我们重新设计整个服务，以使用Queue/Worker类型框架代替，理想情况下，端点一次可以处理一个工作负载，并且是无状态的.

队列将由服务(Jenkins?Lambda?手动?)填充，然后从队列中读取第二个服务(理想情况下，也可以根据需要向外扩展).

解决方案

是耦合"对象，我在想的方法，请参见此处:

您可能会想实现直接耦合，例如允许Lambda 1使用AWS SDK调用Lambda 2，依此类推.这引入了以下一些问题:

1. 如果Lambda 1正在同步调用Lambda 2，则需要等待后者首先完成.Lambda 1可能不知道Lambda 2也同步调用Lambda 3，Lambda 1现在可能需要等待Lambda 2和3成功完成.Lambda 1可能会超时，因为它需要先等待所有Lambda完成，而且您还需要为每一个Lambda等待时付款.

2. 如果Lambda 3设置了并发限制，并且又被其他服务调用，该怎么办?Lambda 2和3之间的调用将失败，直到再次并发为止.该错误可以一直返回到Lambda 1，但是Lambda 1如何处理该错误?它必须存储S3事件未成功，并且需要重播该事件.

此过程可以重新设计为事件驱动:lambda耦合

这不仅解决了直接耦合方法引入的所有问题，而且还提供了一种在每个Lambda发生错误时重播DLQ的方法.没有消息会丢失或需要存储在外部，并且需求与处理无关.

I've been tasked to look at a service built on AWS Lambda that performs a long-running task of turning VMs on and off. Mind you, I come from the Azure team so I am not familair with the styling or best practices of AWS services.

The approach the original developer has taken is to send the entire workload to one Lambda function and then have that function take a section of the workload and then recursively call itself with the remaining workload until all items are gone (workload = 0).

Pseudo-ish Code:

// Assume this gets sent to a HTTP Lambda endpoint as a whole
let workload = [1, 2, 3, 4, 5, 6, 7, 8]

// The Lambda HTTP endpoint
function Lambda(workload) {
    if (!workload.length) {
        return "No more work!"
    }
    const toDo = workload.splice(0, 2) // get first two items
    doWork(toDo)

    // Then... except it builds a new HTTP request with aws sdk
    Lambda(workload) // 3, 4, 5, 6, 7, 8, etc.
}

This seems highly inefficient and unreliable (correct me if I am wrong). There is a lot of state being stored in this process and in my opinion that creates a lot of failure points.

My plan is to suggest we re-engineer the entire service to use a Queue/Worker type framework instead, where ideally the endpoint would handle one workload at a time, and be stateless.

The queue would be populated by a service (Jenkins? Lambda? Manually?), then a second service would read from the queue (and ideally scale-out as well, as needed).

解决方案

It's "Coupling" that I was thinking of, see here: https://www.jeffersonfrank.com/insights/aws-lambda-design-considerations

Coupling
Coupling goes beyond Lambda design considerations—it’s more about the system as a whole. Lambdas within a microservice are sometimes tightly coupled, but this is nothing to worry about as long as the data passed between Lambdas within their little black box of a microservice is not over-pure HTTP and isn’t synchronous.

Lambdas shouldn’t be directly coupled to one another in a Request Response fashion, but asynchronously. Consider the scenario when an S3 Event invokes a Lambda function, then that Lambda also needs to call another Lambda within that same microservice and so on.

aws lambda coupling

You might be tempted to implement direct coupling, like allowing Lambda 1 to use the AWS SDK to call Lambda 2 and so on. This introduces some of the following problems:

1. If Lambda 1 is invoking Lambda 2 synchronously, it needs to wait for the latter to be done first. Lambda 1 might not know that Lambda 2 also called Lambda 3 synchronously, and Lambda 1 may now need to wait for both Lambda 2 and 3 to finish successfully. Lambda 1 might timeout as it needs to wait for all the Lambdas to complete first, and you’re also paying for each Lambda while they wait.

2. What if Lambda 3 has a concurrency limit set and is also called by another service? The call between Lambda 2 and 3 will fail until it has concurrency again. The error can be returned to all the way back to Lambda 1 but what does Lambda 1 then do with the error? It has to store that the S3 event was unsuccessful and that it needs to replay it.

This process can be redesigned to be event-driven: lambda coupling

Not only is this the solution to all the problems introduced by the direct coupling method, but it also provides a method of replaying the DLQ if an error occurred for each Lambda. No message will be lost or need to be stored externally, and the demand is decoupled from the processing.

这篇关于递归AWS Lambda函数调用-最佳实践的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！