任务之间的气流延迟 [英] Airflow latency between tasks

问题描述

如您在图像中看到的：气流使任务之间的时间过长执行？
几乎占DAG执行时间的30％。
我已将 airflow.cfg 文件更改为：

As you can see in the image : airflow is making too much time between tasks execution ? it almost represents 30% of the DAG execution time. I've changed the airflow.cfg file to:

job_heartbeat_sec = 1 
scheduler_heartbeat_sec = 1

但我的延迟率仍然相同。

but I still have the same latency rate.

为什么会这样？

推荐答案

设计。例如，我使用Airflow执行大型工作流程，其中某些任务可能会花费很长时间。气流并不是要执行需要几秒钟才能完成的任务，它当然可以用于执行任务，但不一定是最合适的工具。

It is by design. For instance I use Airflow to perform large workflows where some tasks can take a really long time. Airflow is not meant for tasks that will take seconds to execute, it can be used for that of course but might not be the most suitable tool.

另外，您可能想尝试增加调度程序的线程数：

Additionally you might want to try to increase the number of threads of the scheduler:

   [scheduler]
   max_threads = 4

这也可以通过设置环境变量来完成：

This can alternatively be done by setting the environment variable:

AIRFLOW__SCHEDULER__MAX_THREADS=4

但是，不要指望延迟会减少那么多。

However do not count on the latency to decrease that much.

这篇关于任务之间的气流延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！