Airflow 1.9.0正在排队但没有启动任务 [英] Airflow 1.9.0 is queuing but not launching tasks

问题描述

气流随机不运行排队的任务，某些任务甚至没有获得排队状态。我一直在调度程序日志中看到以下内容

Airflow is randomly not running queued tasks some tasks dont even get queued status. I keep seeing below in the scheduler logs

 [2018-02-28 02:24:58,780] {jobs.py:1077} INFO - No tasks to consider for execution.

我确实看到数据库中没有状态或处于排队状态但从未启动的任务。

I do see tasks in database that either have no status or queued status but they never get started.

气流设置正在运行 https：// github。 com / puckel / docker-airflow 在带有Redis的ECS上。有4个调度程序线程和4个Celery worker任务。对于未运行的任务，将鼠标悬停在任务图标运算符上时，任务将处于排队状态（灰色图标），并且任务详细信息显示为：

The airflow setup is running https://github.com/puckel/docker-airflow on ECS with Redis. There are 4 scheduler threads and 4 Celery worker tasks. For the tasks that are not running are showing in queued state (grey icon) when hovering over the task icon operator is null and task details says:

    All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless:- The scheduler is down or under heavy load

调度程序上的指标未显示重负载。 dag非常简单，只有两个独立的任务，它们仅取决于上一次运行。同一任务中还有一些任务被卡住，没有任何状态（白色图标）。

Metrics on scheduler do not show heavy load. The dag is very simple with 2 independent tasks only dependent on last run. There are also tasks in the same dag that are stuck with no status (white icon).

要注意的有趣事情是，当我重新启动调度程序任务时，它会变为运行状态。

推荐答案

设置气流可能有些棘手。

Airflow can be a bit tricky to setup.

• 您是否拥有气流调度程序正在运行？


• 您是否正在运行 airflow网络服务器？


• 您是否已检查要运行的所有DAG都已在Web ui中设置为 On ？


• 完成所有操作您要运行的DAG的开始日期是过去的时间？


• 您要运行的所有DAG的时间表是否正确，如Web ui所示？


• 如果没有其他效果，则可以使用Web ui单击dag，然后单击 Graph View 。现在选择第一个任务，然后单击任务实例。在任务实例详细信息段落中，您将看到DAG等待或未运行的原因。

• Do you have the airflow scheduler running?
• Do you have the airflow webserver running?
• Have you checked that all DAGs you want to run are set to On in the web ui?
• Do all the DAGs you want to run have a start date which is in the past?
• Do all the DAGs you want to run have a proper schedule which is shown in the web ui?
• If nothing else works, you can use the web ui to click on the dag, then on Graph View. Now select the first task and click on Task Instance. In the paragraph Task Instance Details you will see why a DAG is waiting or not running.

例如，我有一个DAG错误地设置为 depends_on_past：是禁止当前实例正确启动。

I've had for instance a DAG which was wrongly set to depends_on_past: True which forbid the current instance to start correctly.

直接在文档中也有大量资源，其中还有一些提示：为什么我的任务没有得到安排？。

Also a great resource directly in the docs, which has a few more hints: Why isn't my task getting scheduled?.

这篇关于Airflow 1.9.0正在排队但没有启动任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！