用于运行 shell 脚本的 Airflow Dataproc 操作符 [英] Airflow Dataproc operator to run shell scripts

问题描述

是否有任何直接的方法可以将 shell 脚本运行到 dataproc 集群中.目前我可以通过 pysparkoperator 运行 shell(它调用另一个 python 文件，然后这个 python 文件调用 shell 脚本).我搜索了很多链接，但到目前为止还没有找到任何直接的方法.

Is there any direct way to run shell scripts into dataproc cluster. currently i can run the shells through pysparkoperator (which calls aonther python file and then this python file calls shell script). I have searched many links but as of now not found any direct way .

如果有人能告诉我最简单的方法，那对我真的很有帮助.

It will be really helpful for me if anybody can tell me the easiest way.

推荐答案

PIG job with sh operator [1]: gcloud dataproc jobs submit pig ... -e 'sh ls'

PIG job with sh operator [1]: gcloud dataproc jobs submit pig ... -e 'sh ls'

不过我很好奇最终目标是什么?为什么要运行 shell 脚本?如果您的意图是执行一次性集群设置，那么您应该使用初始化操作 [2].

I am however curious what the end goal is? Why run shell scripts? If your intent is to perform one-time cluster setup then you should use initialization actions [2].

[1] https://pig.apache.org/docs/r0.9.1/cmds.html#sh

[2] https://cloud.google.com/dataproc/文档/概念/初始化操作

这篇关于用于运行 shell 脚本的 Airflow Dataproc 操作符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！