任何人都具有使用S3事件触发步进功能的经验吗? [英] Anyone has experience with triggering step function with S3 event?
问题描述
我正在学习步进功能，具体地说，我试图弄清楚如何使用S3事件触发状态机执行.我正在阅读这篇文章: https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html .该文档确实提供了有关如何配置内容的粗略指南，但是我仍然不清楚以下问题:
I'm learning about step function and specifically, I'm trying to figure out how to trigger state machine execution with S3 event. I was reading this post: https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html. This documentation does give a rough guide of how to configure stuff but I'm still unclear about the following questions:
-
状态机输入是什么意思?因此，本文档并没有过多地解释输入中每个字段的含义.有人知道在哪里可以找到它的文档吗?例如，此输入中的id字段是什么?
What does the state machine input mean? So this documentation doesn't explain too much about what does each field in the input mean. Does anyone know where to find the documentation for it? For example, what's the id field in this input?
Java lambda如何从输入中检索有用的信息?我看到了有关如何操作在状态机架构(cloudformation或Amazon Statemachine Lamguage)中预定义的输入的文档，但没有有关由s3事件自动生成的输入的文档.
How does a Java lambda retrieve useful information from the input? I saw documentation about how to manipulate input that was predefined in the state machine schema (cloudformation or Amazon Statemachine Lamguage) but not for the input that was auto-generated by a s3 event.
有人在使用状态机+ s3事件之前构建过类似的功能吗?任何想法都将不胜感激.
Does anyone ever built similar functionality before using state machine + s3 event? Any thoughts would be appreciated.
推荐答案
我们有一个相似的任务-通过S3事件启动StepFunctions状态机-进行少量修改.我们想根据上传文件的扩展名启动不同的状态机.
We had a similar task - start StepFunctions state machine by an S3 event - with a small modification. We wanted to start different state machines based on the extension of the uploaded file.
最初，我们遵循的是您所指的相同教程.以StepFunctions状态机为目标的CloudTrail规则.
Initially we have followed the same tutorial that you're referring to. CloudTrail rule with StepFunctions state machine as target.
但是后来我们意识到我们不能真正通过文件扩展名过滤S3事件(至少我们找不到方法).
But later we have realized that we can't really filter S3 events by file extensions (at least we could not find the way).
最后，我们设法以另一种方式解决了该问题:
Finally we've managed to solve it in a different way:
- S3存储桶配置有通知，这些通知会触发特定的S3对象键后缀(如果需要的话，文件扩展名)的某些lambda函数
- Lambda函数获取S3事件作为输入，根据需要对其进行转换，并使用转换后的输入启动StepFunctions步进计算机.
- StepFunctions状态机以lambda函数已创建的输入启动，并照常执行
与CloudTrail解决方案相比，这有点复杂，因为我们部署了附加的lambda函数.但是我们可以根据需要过滤S3事件，并且我们还可以完全控制馈送到状态机的内容.因此，我认为该解决方案比CloudTrail解决方案更加灵活.
This is a bit more complex compared to the CloudTrail solution as we deploy an additional lambda function. But we can filter S3 events as we need and we have also full control of what is being fed to the state machine. So I think this solution is more flexible than the CloudTrail solution.
我现在将分享我们解决方案的一些细节.我将不得不严格削减我们的代码，因此不能保证这将在OOTB上正常进行，但是希望它足以使您理解.
I will now share some details of our solution. I'll have to severely cut our code, so no guarantees that this will work OOTB, but, hopefully it should be enough for you to get the idea.
上传桶
UploadsInboundBucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: !Sub >-
${AWS::AccountId}-uploads-inbound-bucket
NotificationConfiguration:
LambdaConfigurations:
- Function: !GetAtt StartVideoclipStateMachineExecutionFunction.Arn
Event: 's3:ObjectCreated:*'
Filter:
S3Key:
Rules:
- Name: suffix
Value: mp4
- Function: !GetAtt StartVideoStateMachineExecutionFunction.Arn
Event: 's3:ObjectCreated:*'
Filter:
S3Key:
Rules:
- Name: suffix
Value: json
s3:ObjectCreated:*触发两个lambda函数StartVideoclipStateMachineExecutionFunction或StartVideoStateMachineExecutionFunction之一(取决于对象键的后缀).
s3:ObjectCreated:* triggers (depending on the suffix of the object key) one of two lambda functions StartVideoclipStateMachineExecutionFunction or StartVideoStateMachineExecutionFunction.
在此处详细描述了馈给lambda函数的S3事件: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
The S3 event which is fed to the lambda function is described here in great detail: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
Lambda函数仅解析输入，构建状态机输入并启动状态机.
Lambda function simply parses the input, builds state machine input and starts the state machine.
var stepfunction = require('./stepfunction');
var aws = require('aws-sdk');
var parser = require('./parser');
exports.handler = (event, context, callback) => {
var statemachineArn = process.env.statemachine_arn;
var stepfunctions = new aws.StepFunctions();
stepfunction.startExecutionFromS3Event(stepfunctions, parser, statemachineArn , event);
callback(null, event);
};
解析S3事件:
module.exports = {
parseEvent : function(event)
{
return event.Records[0].s3.bucket.arn + '/'+ event.Records[0].s3.object.key;
}
};
启动状态机执行:
module.exports = {
startExecutionFromS3Event : function(stepfunctions, parser, statemachineArn , event)
{
//get affected S3 object from Event
var arn = parser.parseEvent(event);
//Create input for Step
var input = {
"detail" : {
"resources" : [
{
"type": "AWS::S3::Object",
"ARN": arn
}
]
}
};
//start step function execution
var params = {
stateMachineArn: statemachineArn,
input: JSON.stringify(input)
};
stepfunctions.startExecution(params, function (err, data) {
if (err) {
console.log('err while executing step function')
console.log(JSON.stringify(err));
} else {
console.log('started execution of step function')
}
});
}
}
您还需要大量IAM角色才能完成所有这些工作(例如，必须允许lambda函数启动状态机)，但是在这一点上我将省略它.
You also need a ton of IAM roles an permissions to make all of this work (like, lambda function must be allowed to start the state machine), but I'll omit it at this point.
这篇关于任何人都具有使用S3事件触发步进功能的经验吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
