无法识别气流打包的Dag（Zip） [英] Airflow Packaged Dag (Zip) not recognized

问题描述

我试图将带有Dag的存储库打包到一个Zip文件中，如下所示：在文档中。
因此，我遵循了文档中的约定，即将dag保留在zip的根目录中，并且子目录被airflow视为软件包。

I am trying to package my Repository with my Dag in a Zip file like it states here in the documentation. So i have followed the convention in the documentation, which is to keep the dag in the root of the zip, and the sub directories are viewed as packages by airflow.

我的zip文件包含以下内容：

My zip file has the following contents:

$ unzip -l $AIRFLOW_HOME/dags/test_with_zip.zip 
Archive:  /home/arjunc/Tutorials/airflow/dags/test_with_zip.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2018-03-29 17:46   helloworld/
      189  2018-03-29 17:22   helloworld/hello.py
        0  2018-03-29 17:18   helloworld/__init__.py
      461  2018-03-29 17:24   test_with_zip_dag.py
---------                     -------
      650                     4 files

其中 test_with_zip_dag.py 是文件在带有Dag定义的根目录中，如下所示：

Where test_with_zip_dag.py is the file in the root directory with the Dag definitions as follows:

from datetime import datetime

from airflow import DAG
from airflow.operators.python_operator import PythonOperator

from helloworld.hello import HelloWorld


def run():
    return HelloWorld().run()


dag = DAG('test_with_zip', description='Test Dependencies With Zipping',
          schedule_interval='0 12 * * *',
          start_date=datetime(2017, 3, 20), catchup=False)

hello_operator = PythonOperator(task_id='hello_task', python_callable=run, dag=dag)

我已将此zip放在默认的dags目录$ AIRFLOW_HOME / dags中，但是我的dag无法识别！

I have placed this zip in the default dags directory $AIRFLOW_HOME/dags, but my dag isn't recognized!

我在做什么错了？

当我重新启动Web服务器时，该任务 test_with_zip 弹出了，但是它不能运行，因为调度程序似乎无法识别它。我从网络界面收到以下错误：

When I restarted the webserver, the task test_with_zip has popped up, but it is not runnable because the Scheduler doesn't seem to recognize it. I get the following error for it (from the web interface):

此DAG似乎仅存在于本地。主调度程序似乎不知道它的存在。

This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence.

推荐答案

哪个版本的气流在吗？ Airflow 1.8.1在从zip加载das时遇到问题。此问题已在1.8.3中修复。 https://issues.apache.org/jira/browse/AIRFLOW-1357

Which version of airflow are you on? Airflow 1.8.1 had problems with loading dags from zips. This issue was fixed in 1.8.3. https://issues.apache.org/jira/browse/AIRFLOW-1357

我建议您更新到最新版本的Airflow，即1.9.0

I recommend that you should update to the latest version of Airflow ie 1.9.0

这篇关于无法识别气流打包的Dag（Zip）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！