气流“此DAG在Web服务器DagBag对象中不可用”。 [英] Airflow "This DAG isnt available in the webserver DagBag object "

问题描述

当我在dags文件夹中放置新的DAG python脚本时，我可以在DAG UI中查看DAG的新条目，但不会自动启用。最重要的是，它似乎也没有正确加载。我只能在列表右侧单击几次刷新按钮，然后切换列表左侧的打开/关闭按钮才能安排DAG。这些是手动过程，因为即使DAG脚本放在dag文件夹中，我也需要触发一些操作。

when I put a new DAG python script in the dags folder, I can view a new entry of DAG in the DAG UI but it was not enabled automatically. On top of that, it seems does not loaded properly as well. I can only click on the Refresh button few times on the right side of the list and toggle the on/off button on the left side of the list to be able to schedule the DAG. These are manual process as I need to trigger something even though the DAG Script was put inside the dag folder.

有人可以帮助我吗？我错过了什么吗？还是这是气流的正确行为？

Anyone can help me on this ? Did I missed something ? Or this is a correct behavior in airflow ?

顺便说一句，如帖子标题中所述，有一个指示符带有此消息此DAG在

By the way, as mentioned in the post title, there is an indicator with this message "This DAG isn't available in the webserver DagBag object. It shows up in this list because the scheduler marked it as active in the metdata database" tagged with the DAG title before i trigger all this manual process.

推荐答案

这不是您，也不是正确或预期的行为。
这是Airflow当前的错误。
Web服务器正在以无法真正按预期使用的方式来缓存DagBag。

It is not you nor it is correct or expected behavior. It is a current 'bug' with Airflow. The web server is caching the DagBag in a way that you cannot really use it as expected.

尝试删除DagBag缓存Web服务器仍保留在官方 TODO 作为路线图的一部分，表明该错误可能尚未完全解决，但是这里有一些有关如何进行的建议：

"Attempt removing DagBag caching for the web server" remains on the official TODO as part of the roadmap, indicating that this bug may not yet be fully resolved, but here are some suggestions on how to proceed:

在气流v1.9之前，当dag由导入实例化文件的函数实例化时，会发生这种情况。也就是说：当生成器或工厂模式。在github 2 和JIRA 3 导致修复程序在airflow v1.9中发布。

Prior to airflow v1.9 this occurs when a dag is instantiated by a function which is imported into the file where instantiation happens. That is: when a builder or factory pattern is used. Some reports of this issue on github 2 and JIRA 3 led to a fix released with in airflow v1.9.

如果您使用的是较旧版本的气流，请不要使用构建器功能。

If you are using an older version of airflow, don't use builder functions.

如Dmitri所建议的那样，运行气流回填'< dag_id>' -s'< date>'-e'< date>'有时可以有所帮助。此后，您可能会得到Priyank指出的（非）问题，但这是预期的行为（状态：已暂停还是未暂停），具体取决于安装中的配置。

As Dmitri suggests, running airflow backfill '<dag_id>' -s '<date>' -e '<date>' for the same start and end date can sometimes help. Thereafter you may end up with the (non)-issue that Priyank points, but that is expected behavior (state: paused or not) depending on the configuration you have in your installation.

这篇关于气流“此DAG在Web服务器DagBag对象中不可用”。的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！