Apache Airflow DAG无法导入本地模块 [英] Apache Airflow DAG cannot import local module

问题描述

我似乎不明白如何将模块导入到Apache airflow DAG定义文件中。我希望这样做是为了能够创建一个库，例如，它使声明具有相似设置的任务不再那么冗长。

I do not seem to understand how to import modules into an apache airflow DAG definition file. I would want to do this to be able to create a library which makes declaring tasks with similar settings less verbose, for instance.

这是我能想到的最简单的示例可以解决该问题：我修改了气流教程（ https://airflow.apache.org/tutorial。 html＃recap ）以简单地导入模块并从该模块运行定义。像这样：

Here is the simplest example I can think of that replicates the issue: I modified the airflow tutorial (https://airflow.apache.org/tutorial.html#recap) to simply import a module and run a definition from that module. Like so:

目录结构：

- dags/
-- __init__.py
-- lib.py
-- tutorial.py

tutorial.py：

tutorial.py:

"""
Code that goes along with the Airflow located at:
http://airflow.readthedocs.org/en/latest/tutorial.html
"""
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta

# Here is my added import
from lib import print_double

# And my usage of the imported def
print_double(2)

## -- snip, because this is just the tutorial code, 
## i.e., some standard DAG defintion stuff --

print_double 只是一个简单的def，它将您输入的任何内容乘以2，然后打印结果，但是显然那甚至都不重要，因为这是一个导入问题。

print_double is just a simple def which multiplies whatever input you give it by 2, and prints the result, but obviously that doesn't even matter because this is an import issue.

我能够运行 air根据教程文档成功进行了流测试教程print_date 2015-06-01 -dag运行，并且print_double成功。 4 将按预期方式打印到控制台。一切似乎都很好。

I am able to run airflow test tutorial print_date 2015-06-01 as per the tutorial docs successfully - the dag runs, and moreover the print_double succeeds. 4 is printed to the console, as expected. All appears well.

然后我进入Web UI，并受到损坏的DAG的问候：[/ home / airflow / airflow / dags / tutorial .py]没有名为'lib'的模块。取消暂停dag并尝试使用UI进行手动运行会导致状态为正在运行，但它永远不会成功或失败。它只是永远处于运行状态。我可以根据需要排队，但他们都只能处于运行状态。

Then I go the web UI, and am greeted by Broken DAG: [/home/airflow/airflow/dags/tutorial.py] No module named 'lib'. Unpausing the dag and attempting a manual run using the UI causes a "running" status, but it never succeeds or fails. It just sits on "running" forever. I can queue up as many as I'd like, but they'll all just sit on "running" status.

我检查了气流记录，然后

I've checked the airflow logs, and don't see any useful debug information there.

那我想念的是什么？

推荐答案

再次添加sys路径对我有用

Adding the sys path again worked for me,

import sys
sys.path.insert(0,os.path.abspath(os.path.dirname(__file__)))

这篇关于Apache Airflow DAG无法导入本地模块的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！