“|"有什么作用?和“>>"在 Apache Beam 中意味着什么? [英] What do the "|" and ">>" means in Apache Beam?

问题描述

我正在尝试了解 Apache Beam.我正在关注编程指南，在一个例子中，他们说谈论 下面的代码示例使用 CoGroupByKey 连接两个 PCollections，然后使用 ParDo 来消费结果.然后，代码使用标签从每个集合中查找和格式化数据..

I'm trying to understand Apache Beam. I was following the programming guide and in one example, they say talk about The following code example joins the two PCollections with CoGroupByKey, followed by a ParDo to consume the result. Then, the code uses tags to look up and format data from each collection..

我很惊讶，因为我在任何时候都没有看到 ParDo 操作，所以我开始怀疑 | 是否真的是 ParDo.代码如下所示:

I was quite surprised, because I didn't saw at any point a ParDo operation, so I started to wondering if the | was actually the ParDo. The code looks like this:

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

emails_list = [
    ('amy', '[e-mail]'),
    ('carl', '[e-mail]'),
    ('julia', '[e-mail]'),
    ('carl', '[e-mail]'),
]
phones_list = [
    ('amy', '111-222-3333'),
    ('james', '222-333-4444'),
    ('amy', '333-444-5555'),
    ('carl', '444-555-6666'),
]

pipeline_options = PipelineOptions()
with beam.Pipeline(options=pipeline_options) as p:
    emails = p | 'CreateEmails' >> beam.Create(emails_list)
    phones = p | 'CreatePhones' >> beam.Create(phones_list)
    results = ({'emails': emails, 'phones': phones} | beam.CoGroupByKey())
    
    def join_info(name_info):
        (name, info) = name_info
        return '%s; %s; %s' %\
      (name, sorted(info['emails']), sorted(info['phones']))

    contact_lines = results | beam.Map(join_info)

我注意到 emails 和 phones 在管道开始时被读取，所以我猜它们都是不同的 PCollections， 对?但是 ParDo 在哪里执行?|"有什么作用?和>>"实际上是什么意思?我怎么能看到这个的实际输出?join_info 函数、emails_list 和 phones_list 是否在 DAG 之外定义是否重要?

I do notice that emails and phones are read at the start of the pipeline, so I guess that both of them are different PCollections, right? But where is the ParDo executed? What do the "|" and ">>" actually means? And how I can see the actual output of this? Does it matter if the join_info function, the emails_list and phones_list are defined outside the DAG?

推荐答案

| 表示步骤之间的分离，这是(使用 p as Pbegin): p |ReadFromText(..) |ParDo(..) |GroupByKey().

The | represents a separation between steps, this is (using p as Pbegin): p | ReadFromText(..) | ParDo(..) | GroupByKey().

你也可以在|之前引用其他PCollections:

You can also reference other PCollections before |:

read = p  | ReadFromText(..)
kvs = read | ParDo(..)
gbk = kvs | GroupByKey()

这相当于之前的管道:p |ReadFromText(..) |ParDo(..) |GroupByKey()

>> 用在 | 和 PTransform 之间来命名步骤:p |ReadFromText(..) |键值">>ParDo(..) |GroupByKey()

The >> are used between | and the PTransform to name the steps: p | ReadFromText(..) | "to key value" >> ParDo(..) | GroupByKey()

这篇关于“|"有什么作用?和“>>"在 Apache Beam 中意味着什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！