如何使用"add_value_provider_argument"初始化运行时参数? [英] How to use 'add_value_provider_argument' to initialise runtime parameter?

问题描述

以官方文档创建模板"为例: https://cloud.google.com/dataflow/docs/templates/creating-模板

Take the official document 'Creating Templates' as an example: https://cloud.google.com/dataflow/docs/templates/creating-templates

class WordcountOptions(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
  # Use add_value_provider_argument for arguments to be templatable
  # Use add_argument as usual for non-templatable arguments
  parser.add_value_provider_argument(
      '--input',
      default='gs://dataflow-samples/shakespeare/kinglear.txt',
      help='Path of the file to read from')
  parser.add_argument(
      '--output',
      required=True,
      help='Output file to write results to.')

pipeline_options = PipelineOptions(['--output', 'some/output_path'])
p = beam.Pipeline(options=pipeline_options)
wordcount_options = pipeline_options.view_as(WordcountOptions)
lines = p | 'read' >> ReadFromText(wordcount_options.input)

wordcount_options.input 是一个RuntimeValueProvider.我想使用在运行时指定的值(执行模板)，所以我需要使用 wordcount_options.input.value .但是，创建模板时，它没有属性值".它仅具有"default_value".我尝试在创建模板时指定一个值(以便我现在和以后都可以使用它)，但是无论我在运行时指定什么值，它都只使用创建模板时指定的先前值.

wordcount_options.input is a RuntimeValueProvider. I want to use the value specified at runtime(executing the template), so I need to use wordcount_options.input.value. However, it does not have attribute 'value' when creating the template. It only has 'default_value' instead. I try to specify a value when creating the template(so that I can use it now and later), but no mater what value I specify at runtime, it only use the previous value that I specified when creating the template.

(基本上，我的输入是一个泡菜文件，因此我不能直接使用 wordcount_options.input .)

(Basically, my input is a pickle file so I can not use wordcount_options.input directly.)

推荐答案

链接示例的下面是文档显示在 ValueProvider 参数上使用 .get()方法检索运行时值.

The documentation shows using the .get() method on the ValueProvider parameter to retrieve the runtime value.

请注意，该值不能在管道构建期间使用，因为尚未从模板注入该值.您只应在诸如 DoFn.process()之类的运行时方法内部调用 ValueProvider.get().

Note that the value cannot be used during pipeline construction, since it hasn't been injected from the template. You should only call ValueProvider.get() inside of runtime methods such as DoFn.process().

这篇关于如何使用"add_value_provider_argument"初始化运行时参数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！