有没有办法暂停气流 DagRun? [英] Is there a way to pause an airflow DagRun?

问题描述

有没有办法在 Airflow 中暂停特定的 DagRun?

Is there a way to pause a specific DagRun within Airflow?

我希望能够同时执行多个 DAG 运行，并且我希望能够在某些点单独暂停这些运行.

I want to be able to have multiple, simultaneous executing runs of a single DAG, and I want to be able to pause those runs individually at certain points.

取消暂停/暂停功能似乎只在 DAG 级别工作，并暂停/取消所有 DagRuns(对于该 DAG)的执行.

The unpause/pause function seems to work only at the DAG level and pauses/unpasses all DagRuns (for that DAG) from executing.

我希望能够做到这一点，因为我想要一些长时间运行的异步任务，而且我不想占用运行无限传感器的工作人员，所以我想创建一个任务暂停 dag 和其他一些操作(例如 API 调用)将取消暂停 dag 运行.

I want to be able to do this because I'd like to have some long-running async tasks and I don't want to take up a worker that's running an infinite sensor, so I'd like to create a task that pauses the dag and some other operation ( such as an API call ) will unpause the dag run.

推荐答案

如果这是关于传感器，那么你很幸运，因为该解决方案是在 1.10.2 版本中实现的:https://issues.apache.org/jira/browse/AIRFLOW-2747.

If this is about sensors, you are in luck because the solution was implemented in version 1.10.2: https://issues.apache.org/jira/browse/AIRFLOW-2747.

:param mode: How the sensor operates.
    Options are: ``{ poke | reschedule }``, default is ``poke``.
    When set to ``poke`` the sensor is taking up a worker slot for its
    whole execution time and sleeps between pokes. Use this mode if the
    expected runtime of the sensor is short or if a short poke interval
    is requried.
    When set to ``reschedule`` the sensor task frees the worker slot when
    the criteria is not yet met and it's rescheduled at a later time. Use
    this mode if the expected time until the criteria is met is. The poke
    interval should be more than one minute to prevent too much load on
    the scheduler.

来源:https://github.com/apache/airflow/blob/e62ad5333cbb56ae0f2001f0f79008a21c41a983/airflow/sensors/base_sensor_operator.py#L46

这篇关于有没有办法暂停气流 DagRun?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！