无论如何，在数据流管道中共享状态变量吗? [英] Is there anyway to share stateful variables in dataflow pipeline?

问题描述

我正在使用python建立数据流管道.
我想在流水线转换和工作节点之间共享全局变量，例如全局变量(跨多个工作器).
有什么方法可以支持这一点吗?提前谢谢

I'm making dataflow pipeline with python.
I want to share global variables across pipeline transform and across worker nodes like global variables (across multiple workers).
Is there any way to support this? thanx in advance

推荐答案

状态处理可能用于在特定节点的工作程序之间共享状态(尽管不能在转换之间共享): https://beam.apache.org/blog/2017/02/13/stateful-processing.html

Stateful processing may be of use for sharing state between workers of a specific node (would not be able to share between transforms though): https://beam.apache.org/blog/2017/02/13/stateful-processing.html

这篇关于无论如何，在数据流管道中共享状态变量吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！