Airflow:如何通过 docker-compose.yml 安装 pip 包? [英] Airflow: how to get pip packages installed via their docker-compose.yml?

问题描述

好吧，我可能很愚蠢，但无论如何；如何通过 气流?

OK, I am probably very stupid but anyways; How can I install additional pip packages via the docker-compose file of airflow?

我假设它们应该是获取 requirements.txt 或其他内容的标准功能.在检查他们的 repo 时，我确实看到了一些像 ADDITIONAL_PYTHON_DEPS 这样的 ENV 变量暗示我这应该是可能的，但是在 docker-compose 文件中设置这些变量实际上并没有安装库的.

I am assuming that their should be a standard functionality to pick up a requirements.txt or something. When inspecting their repo, I do see some ENV variables like ADDITIONAL_PYTHON_DEPS that hint me that this should be possible, but setting these in the docker-compose file doesn't actually install the library's.

version: '3'
x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.1}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'
    ADDITIONAL_PYTHON_DEPS: python-bitvavo-api

volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./requirements.txt:/requirements.txt

显然我的 docker 经验非常有限，但我错过了什么?

Obviously my docker experience is very limited but what am I missing?

推荐答案

Airflow 文档 此处.根据您的要求，这可能就像在创建新 Dockerfile 时使用 From 指令扩展原始图像一样简单，或者您可能需要自定义图像以满足您的需要.

There is a pretty detailed guide on how to achieve what you are looking for on the Airflow docs here. Depending on your requirements, this may be as easy as extending the original image using a From directive while creating a new Dockerfile, or you may need to customize the image to suit your needs.

如果您使用 扩展图像 方法你的新 Dockerfile 将是这样的:

If you go with the Extending the image approach your new Dockerfile will be something like this:

FROM apache/airflow:2.0.1
USER root
RUN apt-get update 
  && apt-get install -y --no-install-recommends 
         build-essential my-awesome-apt-dependency-to-add 
  && apt-get autoremove -yqq --purge 
  && apt-get clean 
  && rm -rf /var/lib/apt/lists/*
USER airflow
RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add

然后你可以在 docker-compose 文件中添加这样的东西:

Then you could just add something like these to the docker-compose file:

...
version: "3"
x-airflow-common: &airflow-common
  build: . # this is optional
  image: ${AIRFLOW_IMAGE_NAME:-the_name_of_your_extended_image
  ...
...

最后，构建您的图像并使用 compose 重新打开所有内容.尝试文档以获取详细信息或完整说明.希望对你有用！

Finally, build your image and turn everything back on using compose. Try the docs for details or a full explanation. Hope that works for you!

这篇关于Airflow:如何通过 docker-compose.yml 安装 pip 包?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！