如何从Glue Dev Endpoint运行胶水脚本 [英] How to run glue script from Glue Dev Endpoint
问题描述
我在编辑器中写了一个胶水脚本(test.py).我连接到了胶开发端点，并将脚本复制到了端点，或者可以将其存储在S3存储桶中.基本上，粘合端点是一个EMR群集，现在如何从dev端点终端运行脚本?我可以使用spark-submit并运行它吗?
I have a glue script (test.py) written say in a editor. I connected to glue dev endpoint and copied the script to endpoint or I can store in S3 bucket. Basically glue endpoint is an EMR cluster, now how can I run the script from the dev endpoint terminal? Can I use spark-submit and run it ?
我知道我们可以从胶水控制台运行它，但是更感兴趣的是知道我是否可以从胶水端点终端运行它.
I know we can run it from glue console,but more interested to know if I can run it from glue end point terminal.
推荐答案
您不需要笔记本.您可以SSH到dev端点并使用gluepython解释器(不是普通的python)运行它.
You don't need a notebook; you can ssh to the dev endpoint and run it with the gluepython interpreter (not plain python).
例如
radix@localhost:~$ DEV_ENDPOINT=glue@ec2-w-x-y-z.compute-1.amazonaws.com
radix@localhost:~$ scp myscript.py $DEV_ENDPOINT:/home/glue/myscript.py
radix@localhost:~$ ssh -i {private-key} $DEV_ENDPOINT
...
[glue@ip-w-x-y-z ~]$ gluepython myscript.py
您也可以直接运行脚本，而无需使用ssh获得交互式shell(当然，在使用scp或其他命令上传脚本之后):
You can also run the script directly without getting an interactive shell with ssh (of course, after uploading the script with scp or whatever):
radix@localhost:~$ ssh -i {private-key} $DEV_ENDPOINT gluepython myscript.py
如果这是使用Job类的脚本(如自动生成的Python脚本一样)，则可能需要传递--JOB_NAME和--TempDir参数.
If this is a script that uses the Job class (as the auto-generated Python scripts do), you may need to pass --JOB_NAME and --TempDir parameters.
这篇关于如何从Glue Dev Endpoint运行胶水脚本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
