在Python Apache Beam中使用值提供者参数的方式 [英] Ways of using value provider parameter in Python Apache Beam
问题描述
现在,我可以使用ParDo来获取类中的RunTime值,还有其他方法可以像在函数中那样使用运行时参数吗?
Right now I'm just able to grab the RunTime value inside a class using a ParDo, is there another way to get to use the runtime parameter like in my functions?
这是我现在得到的代码:
This is the code I got right now:
class UserOptions(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
parser.add_value_provider_argument('--firestore_document',default='')
def run(argv=None):
parser = argparse.ArgumentParser()
pipeline_options = PipelineOptions()
user_options = pipeline_options.view_as(UserOptions)
pipeline_options.view_as(SetupOptions).save_main_session = True
with beam.Pipeline(options=pipeline_options) as p:
rows = (p
| 'Create inputs' >> beam.Create([''])
| 'Call Firestore' >> beam.ParDo(
CallFirestore(user_options.firestore_document))
| 'Read DB2' >> beam.Map(ReadDB2))
我希望user_options.firestore_document在无需执行ParDo的情况下也可用于其他功能
I want the user_options.firestore_document to be usable in other functions without having to do a ParDo
推荐答案
使用值提供者的唯一方法是在ParDos和Combines中.无法在创建中传递值提供者,但是您可以定义一个DoFn,该DoFn返回在构造函数中传递给它的值提供者:
The only way in which you can use value providers are in ParDos, and Combines. It is not possible to pass a value provider in a create, but you can define a DoFn that returns the value provider you pass to it in the constructor:
class OutputValueProviderFn(beam.DoFn):
def __init__(self, vp):
self.vp = vp
def process(self, unused_elm):
yield self.vp.get()
在您的管道中,您将执行以下操作:
And in your pipeline, you would do the following:
user_options = pipeline_options.view_as(UserOptions)
with beam.Pipeline(options=pipeline_options) as p:
my_value_provided_pcoll = (
p
| beam.Create([None])
| beam.ParDo(OutputValueProviderFn(user_options.firestore_document))
那样,您将不会在Create中使用它,因为它不可能,但是您仍然可以在PCollection中获得它.
That way you wouldn't use it in a Create, as it's not possible, but you could still get it in a PCollection.
这篇关于在Python Apache Beam中使用值提供者参数的方式的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!