Airflow:如何通过docker-compose.yml安装pip软件包? [英] Airflow: how to get pip packages installed via their docker-compose.yml?

问题描述

好吧，我可能很愚蠢，但无论如何；如何通过

OK, I am probably very stupid but anyways; How can I install additional pip packages via the docker-compose file of airflow?

我假设它们应该是获取 requirements.txt 之类的标准功能.当检查他们的 repo 时，我确实看到了一些ENV变量，例如 ADDITIONAL_PYTHON_DEPS 提示我这是可能的，但是在docker-compose文件中设置这些变量实际上并不会安装该库的变量.

I am assuming that their should be a standard functionality to pick up a requirements.txt or something. When inspecting their repo, I do see some ENV variables like ADDITIONAL_PYTHON_DEPS that hint me that this should be possible, but setting these in the docker-compose file doesn't actually install the library's.

version: '3'
x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.1}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'
    ADDITIONAL_PYTHON_DEPS: python-bitvavo-api

volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./requirements.txt:/requirements.txt

显然我的docker经验非常有限，但是我想念的是什么?

Obviously my docker experience is very limited but what am I missing?

推荐答案

关于如何实现您在Airflow文档上寻找的内容的非常详细的指南

There is a pretty detailed guide on how to achieve what you are looking for on the Airflow docs here. Depending on your requirements, this may be as easy as extending the original image using a From directive while creating a new Dockerfile, or you may need to customize the image to suit your needs.

如果您使用

If you go with the Extending the image approach your new Dockerfile will be something like this:

FROM apache/airflow:2.0.1
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         build-essential my-awesome-apt-dependency-to-add \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add

然后，您可以将以下内容添加到docker-compose文件中:

Then you could just add something like these to the docker-compose file:

...
version: "3"
x-airflow-common: &airflow-common
  build: . # this is optional
  image: ${AIRFLOW_IMAGE_NAME:-the_name_of_your_extended_image
  ...
...

最后，构建图像并使用compose重新打开所有内容.尝试文档以获取详细信息或完整说明.希望对您有用！

Finally, build your image and turn everything back on using compose. Try the docs for details or a full explanation. Hope that works for you!

这篇关于Airflow:如何通过docker-compose.yml安装pip软件包?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！