AWS Glue自动作业创建 [英] AWS Glue automatic job creation

问题描述

我有可以在AWS GLUE中运行的pyspark脚本.但是每次我从UI创建作业并将代码复制到作业中时，无论如何，我都可以从s3存储桶中的文件中自动创建作业. (我拥有将在运行时使用的所有库和粘合上下文)

I have pyspark script which I can run in AWS GLUE. But everytime I am creating job from UI and copying my code to the job .Is there anyway I can automatically create job from my file in s3 bucket. (I have all the library and glue context which will be used while running )

推荐答案

另一种替代方法是使用 AWS CloudFormation .您可以在模板文件中定义要创建的所有AWS资源(不仅是Glue作业)，然后根据需要从

Another alternative is to use AWS CloudFormation. You can define all AWS resources you want to create (not only Glue jobs) in a template file and then update stack whenever you need from AWS Console or using cli.

胶水作业的模板

看起来像这样:

  MyJob:
    Type: AWS::Glue::Job
    Properties:
      Command:
        Name: glueetl
        ScriptLocation: "s3://aws-glue-scripts//your-script-file.py"
      DefaultArguments:
        "--job-bookmark-option": "job-bookmark-enable"
      ExecutionProperty:
        MaxConcurrentRuns: 2
      MaxRetries: 0
      Name: cf-job1
      Role: !Ref MyJobRole # reference to a Role resource which is not presented here

这篇关于AWS Glue自动作业创建的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！