操作员之间的气流和数据传输 [英] Airflow and data transfer between operators

问题描述

我是气流的新手，对 Airflow 及其处理器有疑问.当一个处理器产生一个输出时，这个输出如何在输入中移动到下一个处理器?有一个名为 nifi 的软件，它将中间输出存储到流文件中，afaik 在气流中没有这样的东西.那么这是怎么发生的呢?

I'm new to airflow and have a question about Airflow and its processors. When a processor produces an output, how this output is moved in input to the next processor ? There is a software called nifi which stores intermediate outputs into flowfiles, afaik there is nothing like this in airflow. So how does this happen?

提前致谢.

推荐答案

Airflow 使用 Xcoms 在运算符之间传递数据.

Airflow uses Xcoms to pass data between operators.

如果流程是操作员 A -> 操作员 B，则操作员 A 必须将值推送"到 xcom，如果操作员 B 想要读取它，则必须从 A 中拉出"这个值.

If the flow is operator A -> operator B, then operator A must "push" a value to xcom, and operator B must "pull" this value from A if it wants to read it.

A 下游的任何运算符都可以通过以下方式访问推送到 Xcom 的任何值 A:

Any operators downstream from A have access to any values A pushed to Xcom via:

value = context['task_instance'].xcom_pull(task_ids='operator_a', key='key_name') 

操作员 A 会像这样推送这个值:

And operator A would push this value like this:

context['task_instance'].xcom_push(key_name,value,context['execution_date'])

这篇关于操作员之间的气流和数据传输的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！