气流计划程序无法安排后续任务 [英] Airflow scheduler is slow to schedule subsequent tasks
问题描述
当我尝试在Airflow 1.8.0中运行DAG时，我发现从完成前任任务的时间到选择执行后续任务的时间之间要花费很多时间（通常会增加执行时间）单个任务）。顺序执行器，本地执行器和Celery执行器的情况相同。有什么方法可以减少提到的间接费用时间？ （就像airflow.cfg中可以加速DAG执行的任何参数一样？）
甘特图已添加以供参考：
When I try to run a DAG in Airflow 1.8.0 I find that it takes a lot of time between the time of completion predecessor task and the time at which the successor task is picked up for execution (usually greater the execution times of individual tasks). The same is the scenario for Sequential, Local and Celery Executors. Is there a way to lessen the overhead time mentioned? (like any parameters in airflow.cfg that can speed up the DAG execution?) Gantt chart has been added for reference:
推荐答案
正如尼克所说，气流不是实时工具。任务是按计划安排和执行的，但是下一个任务将永远不会在上一个任务之后立即运行。
As Nick said, Airflow is not a real-time tool. Tasks are scheduled and executed ASAP, but the next Task will never run immediately after the last one.
当每个任务中有〜3个任务的DAG数量超过100个时或具有许多任务（约100个或更多）的Dags，您必须考虑3件事情：
When you have more than ~100 DAGs with ~3 Tasks in each one or Dags with many Tasks (~100 or more), you have to consider 3 things:
- 增加DagFileProcessorManager将要使用的线程数用于加载和执行Dag（airflow.cfg）：
[scheduler]
[scheduler]
max_threads = 2
max_threads = 2
max_threads控制选择并执行/终止了多少个DAG（ 参见此处）。
The max_threads controls how many DAGs are picked and executed/terminated (see here).
增加此配置可以减少任务之间的时间。
Increasing this configuration may reduce the time between the Tasks.
- 监视您的气流数据库以查看是否存在任何瓶颈。 Airflow数据库用于管理和执行流程：
最近，我们遇到了同样的问题。两次任务之间的时间约为10-15分钟，我们在AWS上使用PostgreSQL。
Recently we were suffering with the same problem. The time between Tasks was ~10-15 minutes, we were using PostgreSQL on AWS.
实例未很好地使用资源；约20 IOPS，约20％的内存和约10％的CPU，但气流非常慢。
The instance was not using the resources very well; ~20 IOPS, 20% of the memory and ~10% of CPU, but Airflow was very slow.
在使用PgHero查看数据库性能后，我们发现，即使在小桌子上使用索引进行查询所花的时间超过一秒钟。
After looking at the database performance using PgHero, we discovered that even a query using an Index on a small table was spending more than one second.
因此，我们增加了数据库的大小，现在Airflow的运行速度与火箭一样快。 ：）
So we increased the Database size, and Airflow is now running as fast as a rocket. :)
- 要获取气流花费在加载Dag上的时间，请运行以下命令：
气流list_dags -r
airflow list_dags -r
DagBag解析时间：7.9497220000000075
DagBag parsing time: 7.9497220000000075
如果DagBag解析时间超过5分钟，则可能是个问题。
If the DagBag parsing time is higher than ~5 minutes, it could be an issue.
所有这些都帮助我们更快地运行了Airflow。我真的建议您升级到1.9版，因为此版本已解决许多性能问题
BTW，我们在生产中使用的是Airflow master ，使用LocalExecutor和PostgreSQL作为元数据数据库。
BTW, we are using the Airflow master in production, with LocalExecutor and PostgreSQL as the metadata database.
这篇关于气流计划程序无法安排后续任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
