如何使用“add_value_provider_argument"来初始化运行时参数? [英] How to use 'add_value_provider_argument' to initialise runtime parameter?

问题描述

以官方文档'Creating Templates'为例:https://cloud.google.com/dataflow/docs/templates/creating-模板

Take the official document 'Creating Templates' as an example: https://cloud.google.com/dataflow/docs/templates/creating-templates

class WordcountOptions(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
  # Use add_value_provider_argument for arguments to be templatable
  # Use add_argument as usual for non-templatable arguments
  parser.add_value_provider_argument(
      '--input',
      default='gs://dataflow-samples/shakespeare/kinglear.txt',
      help='Path of the file to read from')
  parser.add_argument(
      '--output',
      required=True,
      help='Output file to write results to.')

pipeline_options = PipelineOptions(['--output', 'some/output_path'])
p = beam.Pipeline(options=pipeline_options)
wordcount_options = pipeline_options.view_as(WordcountOptions)
lines = p | 'read' >> ReadFromText(wordcount_options.input)

wordcount_options.input 是一个 RuntimeValueProvider.我想使用在运行时指定的值(执行模板)，所以我需要使用 wordcount_options.input.value.但是，它在创建模板时没有属性value".它只有default_value".我尝试在创建模板时指定一个值(以便我现在和以后都可以使用它)，但无论我在运行时指定什么值，它都只使用我在创建模板时指定的先前值.

wordcount_options.input is a RuntimeValueProvider. I want to use the value specified at runtime(executing the template), so I need to use wordcount_options.input.value. However, it does not have attribute 'value' when creating the template. It only has 'default_value' instead. I try to specify a value when creating the template(so that I can use it now and later), but no mater what value I specify at runtime, it only use the previous value that I specified when creating the template.

(基本上，我的输入是一个pickle文件，所以我不能直接使用wordcount_options.input.)

(Basically, my input is a pickle file so I can not use wordcount_options.input directly.)

推荐答案

链接示例下方是一个部分 在您的函数中使用 ValueProvider.

Just below the linked example is a section Using ValueProvider in your functions.

文档显示如何使用 ValueProvider 参数上的 .get() 方法来检索运行时值.

The documentation shows using the .get() method on the ValueProvider parameter to retrieve the runtime value.

请注意，在管道构建期间不能使用该值，因为它尚未从模板中注入.您应该只在运行时方法中调用 ValueProvider.get()，例如 DoFn.process().

Note that the value cannot be used during pipeline construction, since it hasn't been injected from the template. You should only call ValueProvider.get() inside of runtime methods such as DoFn.process().

这篇关于如何使用“add_value_provider_argument"来初始化运行时参数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！