AWS Step Functions历史记录事件限制 [英] AWS Step Functions history event limitation

问题描述

我将步函数用于一个大循环，到目前为止没有问题，但是当我的循环超过8000次执行的那一天，我遇到了最大执行历史记录大小"为25000的错误.

有没有历史事件的解决方案吗?

否则，我可以在其中轻松迁移我的步进函数(3 lambda)，因为aws batch将要求我重写很多代码..

非常感谢

解决方案

一种避免25k历史事件限制的方法是在循环中添加一个选择状态，该选择状态接受一个计数器或布尔值并决定退出该循环./p>

您可以在循环外放置一个lambda函数，以启动另一个执行(具有不同的ID).此后，您当前的执行将正常完成，另一个执行将继续执行工作.

请注意，下面的示例中的"LoopProcessor"必须返回变量"$ .breakOutOfLoop"才能退出循环，该循环也必须在循环中的某个位置确定并传递.

根据您的用例，您可能需要重组所传递的数据.例如，如果要处理大量数据，则可能要考虑使用S3对象，并通过状态机执行将ARN作为输入/输出传递.如果您尝试做一个简单的循环，一种简单的方法是添加一个起始偏移量(将其视为全局计数器)，该偏移量作为输入传递到执行中，并且每个LoopProcessor任务都将增加一个计数器(从起始位置开始).偏移量作为初始值).这类似于分页解决方案.

这里是避免25k历史事件限制的ASL结构的基本示例:

{
  "Comment": "An example looping while avoiding the 25k event history limit.",
  "StartAt": "FirstState",
  "States": {

    "FirstState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "Next": "ChoiceState"
    },

    "ChoiceState": {
      "Type" : "Choice",
      "Choices": [
        {
          "Variable": "$.breakOutOfLoop",
          "BooleanEquals": true,
          "Next": "StartNewExecution"
        }
      ],
      "Default": "LoopProcessor"
    },

    "LoopProcessor": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:ProcessWork",
      "Next": "ChoiceState"
    },

    "StartNewExecution": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:StartNewLooperExecution",
      "Next": "FinalState"
    },

    "FinalState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "End": true
    }
  }
}

希望这会有所帮助！

I use step functions for a big loop, so far no problem, but the day when my loop exceeded 8000 executions I came across the error "Maximum execution history size" which is 25000.

There is there a solution for not having the history events?

Otherwise, where I can easily migrate my step functions (3 lambda) because aws batch will ask me a lot of code rewrite ..

Thanks a lot

解决方案

One approach to avoid the 25k history event limit is to add a choice state in your loop that takes in a counter or boolean and decides to exit the loop.

Outside of the loop you can put a lambda function that starts another execution (with a different id). After this, your current execution completes normally and another execution will continue to do the work.

Please note that the "LoopProcessor" in the example below must return a variable "$.breakOutOfLoop" to break out of the loop, which must also be determined somewhere in your loop and passed through.

Depending on your use case, you may need to restructure the data you pass around. For example, if you are processing a lot of data, you may want to consider using S3 objects and pass the ARN as input/output through the state machine execution. If you are trying to do a simple loop, one easy way would be to add a start offset (think of it as a global counter) that is passed into the execution as input, and each LoopProcessor Task will increment a counter (with the start offset as the initial value). This is similar to pagination solutions.

Here is a basic example of the ASL structure to avoid the 25k history event limit:

{
  "Comment": "An example looping while avoiding the 25k event history limit.",
  "StartAt": "FirstState",
  "States": {

    "FirstState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "Next": "ChoiceState"
    },

    "ChoiceState": {
      "Type" : "Choice",
      "Choices": [
        {
          "Variable": "$.breakOutOfLoop",
          "BooleanEquals": true,
          "Next": "StartNewExecution"
        }
      ],
      "Default": "LoopProcessor"
    },

    "LoopProcessor": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:ProcessWork",
      "Next": "ChoiceState"
    },

    "StartNewExecution": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:StartNewLooperExecution",
      "Next": "FinalState"
    },

    "FinalState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "End": true
    }
  }
}

Hope this helps!

这篇关于AWS Step Functions历史记录事件限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！