如何在Glue作业中访问AWS Glue工作流的运行属性? [英] How to access run-property of AWS Glue workflow in Glue job?

问题描述

我一直在使用AWS Glue工作流程来编排批处理作业.我们需要通过

解决方案

如果您将python用作Glue作业的编程语言，则可以发出

这将为您提供以下响应，您可以对其进行解析和使用:

  {'RunProperties':{'string':'string'}} 

如果您使用的是scala，则可以使用等效的AWS开发工具包.

I have been working with AWS Glue workflow for orchestrating batch jobs. we need to pass push-down-predicate in order to limit the processing for batch job. When we run Glue jobs alone, we can pass push down predicates as a command line argument at run time (i.e. aws glue start-job-run --job-name foo.scala --arguments --arg1-text ${arg1}..). But when we use glue workflow to execute Glue jobs, it is bit unclear.

When we orchestrate Batch jobs using AWS Glue workflows, we can add run properties while creating workflow.

1. Can I use run properties to pass push down predicate for my Glue Job ?
2. If yes, then how can I define value for the run property (push down predicate) at run time. The reason I want to define value for push down predicate at run time, is because the predicate arbitrarily changes every day. (i.e. run glue-workflow for past 10 days, past 20 days, past 2 days etc.)

I tried:

aws glue start-workflow-run --name workflow-name | jq -r '.RunId '

aws glue put-workflow-run-properties --name workflow-name --run-id "ID" --run-properties --pushdownpredicate="some value"

I am able to see the run property I have passed using put-workflow-run-property

aws glue put-workflow-run-properties --name workflow-name --run-id "ID"

But I am not able to detect "pushdownpredicate" in my Glue Job. Any idea how to access workflow's run property in Glue Job?

解决方案

If you are using python as programming language for your Glue job then you can issue get_workflow_run_properties API call to retrieve the property and use it inside your Glue job.

response = client.get_workflow_run_properties(
    Name='string',
    RunId='string'
)

This will give you below response which you can parse and use it:

{
    'RunProperties': {
        'string': 'string'
    }
}

If you are using scala then you can use equivalent AWS SDK.

这篇关于如何在Glue作业中访问AWS Glue工作流的运行属性?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！