如何使用 Airflow 在不同的机器上运行一个工作流程的不同任务? [英] How can Airflow be used to run distinct tasks of one workflow in separate machines?

问题描述

免责声明:我(还)不是 Airflow 的用户，今天才发现它，我开始探索它是否适合我的用例.

disclaimer: I'm not (yet) a user of Airflow, just found about it today and I'm starting to explore if it may fit my uses cases.

我有一个数据处理工作流，它是多个任务的顺序(非并行)执行.但是，某些任务需要在特定机器上运行.Airflow 可以管理这个吗?此用例的建议实施模型是什么?

I have one data processing workflow that is a sequential (not parallel) execution of multiple tasks. However, some of the tasks need to run on specific machines. Can Airflow manage this? What would be the advised implementation model for this use case?

谢谢.

推荐答案

是的，您可以在 Airflow 中通过 队列.您可以将任务绑定到特定队列.然后对于机器上的每个工作人员，您可以将其设置为仅从选定队列中提取任务.

Yes, you can achieve this in Airflow with queues. You can tie tasks to a specific queue. Then for each worker on a machine, you can set it to only pickup tasks from select queues.

在代码中，它看起来像这样:

In code, it would look like this:

task_1 = BashOperator(
    dag=dag,
    task_id='task_a',
    ...
)

task_2 = PythonOperator(
    dag=dag,
    task_id='task_b',
    queue='special',
    ...
)

注意airflow.cfg中有这个设置:

Note that there is this setting in airflow.cfg:

# Default queue that tasks get assigned to and that worker listen on.
default_queue = default

因此，如果您以这样的方式开始工作:

So if you started your workers with this:

Server A> airflow worker
Server B> airflow worker --queues special
Server C> airflow worker --queues default,special

那么task_1可以被服务器A+C拾取，task_2可以被服务器B+C拾取.

Then task_1 can be picked up by servers A+C and task_2 can be picked up by servers B+C.

这篇关于如何使用 Airflow 在不同的机器上运行一个工作流程的不同任务?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！