如何在 Airflow 中安装软件包? [英] How to install packages in Airflow?

问题描述

我在 Airflow(在 GCP 上)部署了一个 dag，但收到错误没有名为‘scipy’的模块".如何在 Airflow 中安装软件包?

I deployed a dag in Airflow (on GCP) but I receive error "No module named 'scipy'". How do I install packages in Airflow?

我尝试添加一个单独的 DAG 来运行

I've tried adding a separate DAG to run

def pip_install(package):
    subprocess.call([sys.executable, "-m", "pip", "install", package])


def update_packages(**kwargs):
    logging.info(list(sys.modules.keys()))
    for package in PACKAGES:
        pip_install(package)

我试过在 GCP 的 shell 上写 pip3 install scipy;

I've tried writing pip3 install scipy on the shell of GCP;

我尝试将 pip install scipy 添加到映像生成器.

I've tried adding pip install scipy to the image builder.

这些方法都没有任何结果.

None of these approaches had any result.

推荐答案

如果您在 GCP 上使用 Cloud Composer，您应该检查 https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies

If you are using Cloud Composer on GCP, you should check https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies

将 requirements.txt 文件传递​​给 gcloud 命令行工具.在单独的行中使用每个需求说明符格式化文件.

Pass a requirements.txt file to the gcloud command-line tool. Format the file with each requirement specifier on a separate line.

示例 requirements.txt 文件:

Sample requirements.txt file:

scipy>=0.13.3
scikit-learn
nltk[machine_learning]

将 requirements.txt 文件传递​​给 gcloud 命令以设置安装依赖项.

Pass the requirements.txt file to the gcloud command to set your installation dependencies.

gcloud composer environments update ENVIRONMENT-NAME \\
--update-pypi-packages-from-file requirements.txt \\
--location LOCATION

这篇关于如何在 Airflow 中安装软件包?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！