S3 Lambda 在 10 分钟后触发双重调用 [英] S3 Lambda trigger double invocation after exactly 10 minutes

问题描述

我们遇到了由 S3 ObjectCreated-Events 触发的两次 Lambda 调用.这些双重调用发生在恰好第一次调用后 10 分钟，不是第一次尝试完成后 10 分钟，而是第一次调用发生后 10 分钟.原始调用需要 0.1 到 5 秒之间的任何时间.没有调用会导致错误，它们都成功完成.

We are experiencing double Lambda invocations of Lambdas triggered by S3 ObjectCreated-Events. Those double invocations happen exactly 10 minutes after the first invocation, not 10 minutes after the first try is complete, but 10 minutes after the first invocation happened. The original invocation takes anything in the range between 0.1 to 5 seconds. No invocations results in errors, they all complete successfully.

我们知道，例如 SQS 并不能保证消息的精确一次但至少一次传递，并且由于底层分布式系统的结果，我们会接受一些 lambda 表达式被第二次调用.然而，延迟 10 分钟听起来很奇怪.

We are aware of the fact that SQS for example does not guarantee exactly-once but at-least-once delivery of messages and we would accept some of the lambdas getting invoked a second time due to results of the distributed system underneath. A delay of 10 minutes however sounds very weird.

在大约 10k 条消息中，100-200 条导致双重调用.

Of about 10k messages 100-200 result in double invocations.

AWS Support 基本上说10 分钟的等待时间是设计使然，但我们无法告诉您原因"，这根本没有帮助.

The AWS Support basically says "the 10 minute wait time is by design but we cannot tell you why", which is not at all helpful.

• 之前有其他人遇到过这种行为吗?
• 您是如何解决这个问题的，还是您只是忽略了它(我们可以这样做)?
• 一个提议的解决方案是不使用直接的 S3-lambda-触发器，而是让 S3 将其事件放在 SNS 上并订阅一个 Lambda.有这种方法的经验吗?

示例日志:两次调用，相隔 10 分钟，相同的 RequestId

START RequestId:f9b76436-1489-11e7-8586-33e40817cb02 版本:13
2017-03-29 14:14:09 INFO ImageProcessingLambda:104 - 处理 1 条记录

START RequestId: f9b76436-1489-11e7-8586-33e40817cb02 Version: 13
2017-03-29 14:14:09 INFO ImageProcessingLambda:104 - handle 1 records

和

START RequestId:f9b76436-1489-11e7-8586-33e40817cb02 版本:13
2017-03-29 14:24:09 INFO ImageProcessingLambda:104 - 处理 1 条记录

START RequestId: f9b76436-1489-11e7-8586-33e40817cb02 Version: 13
2017-03-29 14:24:09 INFO ImageProcessingLambda:104 - handle 1 records

推荐答案

经过 AWS 支持和其他人的几轮测试以及一些孤立的试运行后，这似乎只是设计使然".不清楚为什么，但它只是发生了.问题既不是 S3 也不是 SQS/SNS，而只是 lambda 调用以及 lambda 服务如何将调用分派到 lambda 实例.

After a couple of rounds with the AWS support and others and a few isolated trial runs it seems like this is simply "by design". It is not clear why, but it simply happens. The problem is neither S3 nor SQS / SNS but simply the lambda invocation and how the lambda service dispatches the invocations to lambda instances.

两次调用发生在所有调用的 1% 到 3% 之间，第一次调用后 10 分钟.令人惊讶的是，甚至有三次(可能是四次)调用的幂率是基本概率，所以基本上是 0.09%，......三次调用发生在第一个调用之后 20 分钟.

The double invocations happen somewhere between 1% and 3% of all invocations, 10 minutes after the first invocation. Surprisingly there are even triple (and probably quadruple) invocations with a rate of powers of the base probability, so basically 0.09%, ... The triple invocations happened 20 minutes after the first one.

如果遇到这种情况，您只需使用您有权访问的任何内容来解决它.例如，我们现在将已处理的实体存储在 Cassandra 中，TTL 为 1 小时，如果实体尚未处理，则仅响应来自 lambda 的消息.两次和三次调用都发生在这一小时的时间范围内.

If you encounter this, you simply have to work around it using whatever you have access to. We for example now store the already processed entities in a Cassandra with a TTL of 1 hour and only responding to messages from the lambda if the entity has not been processed yet. The double and triple invocations all happen within this one hour timeframe.

这篇关于S3 Lambda 在 10 分钟后触发双重调用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！