如果我导入google.cloud.storage，则无法在Google Composer上部署DAG [英] Cannot deploy DAGs on Google Composer if I import google.cloud.storage

问题描述

由于某些原因，如果我在DAG中导入google.cloud.storage，则无法在Google Composer上部署DAG文件.如果我尝试部署此类DAG文件，则该文件不会添加到DagBag中，因此最终在Airflow网站上显示为非链接条目，因此无法使用.此时，通常会显示信息图标:该DAG在Web服务器的DagBag对象中不可用.它显示在此列表中是因为调度程序在元数据数据库中将其标记为活动状态.与实际的语法错误不同，页面顶部没有错误消息.

For some reason I can't deploy DAG files on Google Composer if I import google.cloud.storage in the DAG. If I try to deploy such a DAG file then it doesn't get added to the DagBag so ends up with a non-link entry in the Airflow website and is not usable. At this point there's the usual information icon saying: This DAG isn't available in the web server's DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database. Unlike an actual syntax error there is no error message at the top of the page.

对于是否要导入google.cloud.storage，我已将其归结为准确的解释.甚至我实际上都没有使用它.例如，如果我注释掉存储导入行，则此dag可以很好地工作，如果替换了它，则不会安装在Composer中.有谁知道为什么吗?

I have boiled this down precisely as to whether I import google.cloud.storage or not. Not even whether I actually use it. For example this dag works fine if I comment out the storage import line, does not install in Composer if I replace it. Would anyone have any clue as to why?

import datetime
from airflow import DAG
from google.cloud import storage
from airflow.operators.python_operator import PythonOperator

default_args = {
    'owner': 'Airflow',
    'depends_on_past': False,
    'email': ['[e-mail]'],
    'email_on_failure': True,
    'email_on_retry': True,
    'retries': 1,
    'retry_delay': datetime.timedelta(minutes=5),
    'start_date': datetime.datetime(2017,1,1),
}

def ingest_file(**kwargs):
    status = 'OK'
    return status

# Not scheduled, trigger only
dag = DAG('ingest_test', default_args=default_args, schedule_interval=None)

ingest = PythonOperator(task_id = 'ingest', provide_context = True, 
  python_callable = ingest_file, dag = dag)

推荐答案

如果在DAG或自定义运算符中需要PyPi程序包，则不会收到错误消息，则DAG不会部署.如果要这样做，请确保将所需的所有软件包都安装在Composer环境中.

If you require PyPi packages in your DAG or custom Operators then you don't get an error message, the DAG just doesn't deploy. If you're getting this then make sure all the packages you need are installed in the Composer environment.

请注意，先存在然后不存在的行为仍然存在，但实际上会在不久后稳定下来.

Note, that the behaviour of being present then not present is still there but does actually settle after a short while

这篇关于如果我导入google.cloud.storage，则无法在Google Composer上部署DAG的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！