如何运行简单的气流DAG [英] How to Run a Simple Airflow DAG

问题描述

我对Airflow完全陌生。我想在指定的日期运行一个简单的DAG。我正在努力使开始日期，执行日期和回填之间有所不同。而运行DAG的命令是什么？

I am totally new to Airflow. I would like to run a simple DAG at a specified date. I'm struggling to make difference between the start date, the execution date, and backfilling. And what is the command to run the DAG?

这是我从那以后尝试过的命令：

Here is what I've tried since:

airflow run dag_1 task_1 2017-1-23

第一次运行该命令，任务正确执行，但是当我再次尝试时，它不起作用。

The first time I ran that command, the task executed correctly, but when I tried again it did not work.

这是我运行的另一个命令：

Here is another command I ran:

airflow backfill dag_1 -s 2017-1-23 -e 2017-1-24

我不知道该命令会带来什么。 DAG会每天从23到24执行吗？

I don't know what to expect from this command. Will the DAGs execute every day from 23 to 24?

在运行上述两个命令之前，我已经这样做了：

Before running the two commands above, I did this:

airflow initdb
airflow scheduler 
airflow webserver -p 8085 --debug &

这是我的DAG

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2017, 1, 23, 12),
    'email': ['[e-mail]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'dag_1', default_args=default_args, schedule_interval=timedelta(1))

t1 = BashOperator(
    task_id='create_clients',
    bash_command='Rscript /scripts/Cli.r',
    dag=dag)

t2 = BashOperator(
    task_id='create_operation',
    bash_command='Rscript Operation.r',
    retries=3,
    dag=dag)

t2.set_upstream(t1)

截屏：树状视图

更新 >

airflow run dag_1 task_1 2017-1-23T10:34

推荐答案

如果使用

airflow run dag_1 task_1 2017-1-23

运行已保存并正在运行它再也无能为力，您可以尝试通过强制使其重新运行

The run is saved and running it again won't do anything you can try to re-run it by forcing it

airflow run --force=true dag_1 task_1 2017-1-23

airflow backfill命令将运行当时将执行的所有执行从开始到结束日期指定的时间段。这将取决于您在DAG上设置的时间表，如果将其设置为每小时触发一次，则它应该运行24次，但它也不会重新执行以前执行的运行。

The airflow backfill command will run any executions that would have run in the time period specified from the start to end date. It will depend what schedule you set on the DAG, if you set it to trigger every hour it should run 24 times, but it also won't re-execute previously executed runs.

您可以清除任务，就像它从未运行过

You can clear the task as if it NEVER ran

airflow clear dag_1 -s 2017-1-23 -e 2017-1-24

也请在此处检查cli文档： https://airflow.incubator.apache.org/cli.html

Also check the cli docs here: https://airflow.incubator.apache.org/cli.html

这篇关于如何运行简单的气流DAG的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！