如何管理气流dag之间的python软件包？ [英] How to manage python packages between airflow dags?

问题描述

如果我有多个气流dag，并且它们之间有一些重叠的python软件包依赖项，那么如何保留每个项目部门。解耦？例如。如果我在同一台服务器上有项目A和B，我将使用类似的命令运行它们中的每一个……

If I have multiple airflow dags with some overlapping python package dependencies, how can I keep each of these project deps. decoupled? Eg. if I had project A and B on same server I would run each of them with something like...

source /path/to/virtualenv_a/activate
python script_a.py
deactivate
source /path/to/virtualenv_b/activate
python script_b.py
deactivate

基本上，希望在相同情况下运行dag（例如，每个dag使用python脚本我可能希望单独开发具有重叠的软件包部门（即，只想为一个项目更新软件包时，不必使用软件包更新所有代码）。注意，我一直在使用 BashOperator 运行python任务，例如...

Basically, would like to run dags with the same situation (eg. each dag uses python scripts that have may have overlapping package deps. that I would like to develop separately (ie. not have to update all code using a package when want to update the package just for one project)). Note, I've been using the BashOperator to run python tasks like...

do_stuff = BashOperator(
        task_id='my_task',
        bash_command='python /path/to/script.py'),
        execution_timeout=timedelta(minutes=30),
        dag=dag)

有没有办法这个工作吗？气流是否还有其他最佳实践方法可以使人们解决（或避免）此类问题？

Is there a way to get this working? IS there some other best-practice way that airflow intendeds for people to address (or avoid) these kinds of problems?

推荐答案

您可以使用打包的dag，其中每个dag都与它的依赖项打包在一起
http：/ /airflow.apache.org/concepts.html#packaged-dags

you can use packaged dags, where each dag is packaged with its dependency http://airflow.apache.org/concepts.html#packaged-dags

这篇关于如何管理气流dag之间的python软件包？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！