如何使用 Cloud Composer 在 Compute Engine 虚拟机上安排 Jupyter Notebook? [英] How to use Cloud Composer to schedule Jupyter Notebooks on a Compute Engine VM?

问题描述

我想在每个工作日执行一系列笔记本，每个笔记本都依赖于前一个，我希望使用 Cloud Composer 使整个过程自动化.

I have a series of notebooks I want to execute each weekday, each of which is dependent on the previous one, and I'd like to have this entire process automated with Cloud Composer.

我熟悉 Papermill 库，所以这部分不是问题.我遇到的问题是如何让存储桶中的 DAG 与 Compute Engine 上的虚拟机实例进行通信.

I'm familiar with the Papermill library, so that part isn't an issue. What I am having issues with is how to have the DAG in the bucket communicate with the virtual machine instance on Compute Engine.

感谢任何帮助，谢谢！

推荐答案

Composer 允许您使用 Compute Engine Pyhton API.

Composer allows you to interact with its GCP Virtual Machines through the python operatorby using the Compute Engine Pyhton API.

您可以有一个 启动脚本，它利用 Papermill 进行交互每次启动机器时都使用笔记本(并使用 Composer 来安排何时停止或启动).

You could have a startup script that makes use of Papermill in order to interact with the notebook(s) every time you start your machine (and use Composer in order to schedule when should these be stopped or started).

这对你有用吗?您的用例是否有任何使这不是可行选项的地方?

Would this work for you? Is there anything about your use-case that would make this not a feasible option?

这篇关于如何使用 Cloud Composer 在 Compute Engine 虚拟机上安排 Jupyter Notebook?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！