如何计划作业以在云中执行Python脚本以将数据加载到bigquery中? [英] How to schedule a job to execute Python script in cloud to load data into bigquery?

问题描述

我正在尝试在云中设置计划作业/流程，以使用python脚本将CSV数据从Google存储桶中加载到Bigquery中.我设法阻止了python代码来执行此操作，但不确定我需要在哪里保存此代码，以便可以将该任务作为自动化过程完成，而不必手动运行gsutil命令.

I am trying to setup a schedule job/process in cloud to load csv data into Bigquery from google buckets using a python script. I have manage to get hold off the python code to do this but not sure where do I need to save this code so that this task could be completed as an automated process rather than running the gsutil commands manualy.

推荐答案

Google Compute Engine上可靠的任务计划解决方案Google Cloud Platform 是Google在"google cloud schedule a cron job"上的第一个链接，它提供了较高层次的概述.使用Cron for Python计划任务||适用于Python的App Engine标准环境|第二个链接是Google Cloud Platform ，其中包含分步说明.他们归结为:

Reliable Task Scheduling on Google Compute Engine  |  Solutions  |  Google Cloud Platform, the 1st link in Google on "google cloud schedule a cron job", gives a high-level overview. Scheduling Tasks With Cron for Python  |  App Engine standard environment for Python  |  Google Cloud Platform, the 2nd link, has step-by-step instructions. They boil down to:

• 与您的 app.yaml
一起以特定的格式创建 cron.yaml
• (可选)在开发服务器上对其进行测试
• 使用 appcfg.py update 或 update_cron
将其上传到Google Cloud

• Create cron.yaml in the specificed format alongside your app.yaml
• optionally test it at a development server
• upload it to the Google Cloud with appcfg.py update or update_cron

这篇关于如何计划作业以在云中执行Python脚本以将数据加载到bigquery中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！