气流可以用来执行永无止境的任务吗？ [英] Can airflow be used to run a never ending task?

问题描述

我们是否可以通过将任务/延迟超时设置为无并手动触发其运行，来使用气流dag来定义永无止境的工作（即，具有无条件循环以消耗流数据的任务）？配备气流监测器永无止境的任务会引起问题吗？
谢谢

Can we use an airflow dag to define a never-ending job (ie. a task which has a unconditional loop to consume stream data) by setting the task/dag timeout to None and manually trigger its running? Would having airflow monitor a never ending task cause a problem? Thanks

推荐答案

通过Airflow运行它有点奇怪，但是是的，我认为这不是问题。请注意，如果重新启动运行该作业的工作程序（假设使用CeleryExecutor），则会中断任务，并且如果未设置重试，则需要再次手动将其启动。还要确保DAG设置 execution_timeout = None （默认）。

A bit odd to run this through Airflow, but yeah I don't think that's an issue. Just note that if you restart the worker running the job (assuming CeleryExecutor), you'll interrupt the task and need to kick it off manually again if retries is not set. Also make sure the DAG sets execution_timeout=None (default).

这篇关于气流可以用来执行永无止境的任务吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！