我可以在本地测试AWS Glue代码吗? [英] Can I test AWS Glue code locally?

问题描述

阅读Amazon文档后，我的理解是，运行/测试Glue脚本的唯一方法是将其部署到开发终端节点，并在必要时进行远程调试.同时，如果(Python)代码由多个文件和程序包组成，则除主脚本外的所有文件和程序包都需要压缩.所有这些使我感到Glue不适合任何复杂的ETL任务，因为开发和测试很麻烦.我可以在本地测试我的Spark代码，而不必每次都将代码上传到S3，并且可以在CI服务器上验证测试，而不必支付开发Glue端点的费用.

After reading Amazon docs, my understanding is that the only way to run/test a Glue script is to deploy it to a dev endpoint and debug remotely if necessary. At the same time, if the (Python) code consists of multiple files and packages, all except the main script need to be zipped. All this gives me the feeling that Glue is not suitable for any complex ETL task as development and testing is cumbersome. I could test my Spark code locally without having to upload the code to S3 every time, and verify the tests on a CI server without having to pay for a development Glue endpoint.

推荐答案

最终，截至2019年8月28日，Amazon允许您下载二进制文件和

Eventually, as of Aug 28, 2019, Amazon allows you to download the binaries and

开发，编译，调试和单步执行Glue ETL脚本，并且复杂 在本地使用Scala和Python生成Spark应用程序.

develop, compile, debug, and single-step Glue ETL scripts and complex Spark applications in Scala and Python locally.

查看此链接: 查看全文