Apache Beam:在已构建的管道中跳过步骤 [英] Apache Beam: Skipping steps in an already-constructed pipeline

问题描述

有没有办法有条件地跳过已经构建的管道中的步骤?或者管道构建是否被设计为控制运行哪些步骤的唯一方法?

Is there a way to conditionally skip steps in an already-constructed pipeline? Or is pipeline construction designed to be the only way to control which steps are run?

推荐答案

通常，管道构造控制将执行管道中的哪些转换.

Normally, pipeline construction controls what transformations in a pipeline will be executed.

然而，您可以想象单个输入、多个输出 ParDo 将输入 PCollection 多路复用到输出 PCollection 之一.然后，通过选择将数据传递到哪个输出，您可以动态控制执行哪些步骤——没有任何输入的步骤可能不会被执行和/或它们的执行可能无关紧要.

You can, however, imagine a single input, multiple output ParDo that multiplexes the input PCollection to one of the output PCollections. Then, by choosing which output to pass your data to, you can dynamically control which steps are executed -- steps without any input might not be executed and/or their execution might not matter.

一个相关的功能是参数化管道"或模板管道".这是我们非常感兴趣并正在积极努力的事情.

A related feature is "parameterized pipelines" or "template pipelines". This is something we are very interested in and are actively working on.

这篇关于Apache Beam:在已构建的管道中跳过步骤的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！