从AWS Glue脚本调用存储过程 [英] Calling stored procedure from aws Glue Script

问题描述

完成ETL作业后，在AWS Glue脚本中调用存储过程的最佳方法是什么?

After the ETL Job is done, What is the best way to call stored procedure in AWS Glue script?

我正在使用PySpark从S3中获取数据并存储在登台表中.完成此过程后，需要调用存储过程.此存储过程将数据从登台表加载到适当的MDS表中.

I am using PySpark to fetch the data from S3 and storing in staging table. After this process, need to call a stored procedure. This stored procedure loads data from the staging table into the appropriate MDS tables.

如果在完成ETL作业后必须调用存储过程，最好的方法是什么?如果我考虑使用AWS Lambda，有什么方法可以在ETL之后通知lambda.

If I have to call a Stored Procedure after ETL Job is done, what is the best way? If I consider AWS lambda, is there any way that lambda can be notified after the ETL.

推荐答案

在胶水ETL作业结束时，您可以使用boto sdk触发lambda.由于您正在写入临时表，因此还需要在连接的子网中具有NAT网关才能使boto调用正常工作.此链接上的示例使用boto触发ECS任务，这将是一个类似的过程: https://github.com/full360/glue-sneaql-demo/blob/master/Glue-Job-Script .

You can use the boto sdk to fire off your lambda at the end of your glue ETL job. Since you are writing to a staging table, you will also need to have a NAT Gateway on the connection's subnet for the boto call to work. The example at this link uses boto to fire an ECS task, which would be a similar process: https://github.com/full360/glue-sneaql-demo/blob/master/Glue-Job-Script.

这篇关于从AWS Glue脚本调用存储过程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！