将Google数据流管道的结果写入多个接收器 [英] Writing results of google dataflow pipeline into mulitple sinks

问题描述

我想将Google数据流管道结果写入多个接收器.

I would like to write the Google dataflow pipeline results into multiple sinks.

例如，我想使用TextIO将结果写入Google Cloud Storage，以及将结果写入BigQuery中的表格中.我该怎么办?

Like, I want to write the result using TextIO into Google Cloud Storage as well write the results as a table in BigQuery. How can I do that?

推荐答案

Cloud Dataflow管道的结构是DAG(有向无环图)，可以将多个转换应用于同一个PCollection-写转换不是例外.您可以将多个写入转换应用于结果的PCollection，例如:

The structure of a Cloud Dataflow pipeline is a DAG (directed acyclic graph) and it is allowed to apply multiple transforms to the same PCollection - write transforms are not an exception. You can apply multiple write transforms to the PCollection of your results, for example:

PCollection<Foo> results = p.apply(TextIO.Read.named("ReadFromGCS").from("gs://..."))
                         .apply(...the rest of your pipeline...);
results.apply(TextIO.Write.named("WriteToGCS").to("gs://..."));
results.apply(BigQueryIO.Write.named("WriteToBigQuery").to(...)...);

这篇关于将Google数据流管道的结果写入多个接收器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！