如何从 Google Dataflow 中的 PCollection 中获取元素列表并在管道中使用它来循环写入转换? [英] How to get a list of elements out of a PCollection in Google Dataflow and use it in the pipeline to loop Write Transforms?

问题描述

我将 Google Cloud Dataflow 与 Python SDK 结合使用.

I am using Google Cloud Dataflow with the Python SDK.

我想:

• 从主 PCollection 中获取唯一日期列表
• 遍历该列表中的日期以创建过滤后的 PCollection(每个都有一个唯一的日期)，并将每个过滤后的 PCollection 写入 BigQuery 中时间分区表中的分区.

我怎样才能得到那个列表?在以下组合转换之后，我创建了一个 ListPCollectionView 对象，但我无法迭代该对象:

How can I get that list ? After the following combine transform, I created a ListPCollectionView object but I cannot iterate that object :

class ToUniqueList(beam.CombineFn):

    def create_accumulator(self):
        return []

    def add_input(self, accumulator, element):
        if element not in accumulator:
            accumulator.append(element)
        return accumulator

    def merge_accumulators(self, accumulators):
        return list(set(accumulators))

    def extract_output(self, accumulator):
        return accumulator


def get_list_of_dates(pcoll):

    return (pcoll
            | 'get the list of dates' >> beam.CombineGlobally(ToUniqueList()))

我做错了吗?最好的方法是什么?

Am I doing it all wrong ? What is the best way to do that ?

谢谢.

推荐答案

无法直接获取 PCollection 的内容 - Apache Beam 或 Dataflow 管道更像是应该进行什么处理，PCollection 是计划中的逻辑中间节点，而不是包含数据.主程序组装计划(管道)并启动它.

It is not possible to get the contents of a PCollection directly - an Apache Beam or Dataflow pipeline is more like a query plan of what processing should be done, with PCollection being a logical intermediate node in the plan, rather than containing the data. The main program assembles the plan (pipeline) and kicks it off.

但是，最终您要尝试将数据写入按日期分片的 BigQuery 表.目前仅支持此用例 在 Java SDK 中，并且仅用于流管道.

However, ultimately you're trying to write data to BigQuery tables sharded by date. This use case is currently supported only in the Java SDK and only for streaming pipelines.

对于根据数据将数据写入多个目的地的更一般处理，请遵循 BEAM-92.

For a more general treatment of writing data to multiple destinations depending on the data, follow BEAM-92.

另见创建/写作通过 Google Cloud Dataflow 到 Parititoned BigQuery 表

这篇关于如何从 Google Dataflow 中的 PCollection 中获取元素列表并在管道中使用它来循环写入转换?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！