在模板执行时未接受ValueProvider类型的参数 [英] ValueProvider type parameters not getting honored at the template execution time

问题描述

我试图在执行时传递在TemplateOption类中定义为ValueProvider的BigTable tableId，instanceId和projectId，因为它们是运行时值，但它们不适合新值. pipleine将使用构造管道时定义的旧值执行.我应该进行哪些更改以使其在运行时符合值?

I am trying to pass BigTable tableId, instanceId and projectId which are defined as ValueProvider in the TemplateOption class at the execution time as they are runtime values but they don't get honored with the new values . The pipleine gets executed with the old values which were defined when the pipeline was constructed. What changes should i make so that it honors values at runtime?

Pipeline p = Pipeline.create(options);
com.google.cloud.bigtable.config.BigtableOptions.Builder optionsBuilder =
        new com.google.cloud.bigtable.config.BigtableOptions.Builder().
                setProjectId("my-project");   

PCollection<com.google.bigtable.v2.Row> row = p.apply("filtered read", org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.read().withBigtableOptions(optionsBuilder).withoutValidation().withInstanceId(options.getInstanceId()).withProjectId(options.getProjectId()).withTableId(options.getTableId()));
PCollection<KV<Integer,String>> convertToKV = row.apply(ParDo.of(new ConvertToKV()));  

我的Option类看起来像:-

My Option class looks like :--

@Default.String("my-project")
@Description("The Google Cloud project ID for the Cloud Bigtable instance.")
ValueProvider<String> getProjectId();
void setProjectId(ValueProvider<String> projectId);

@Default.String("my-instance")
@Description("The Google Cloud Bigtable instance ID .")
ValueProvider<String> getInstanceId();
void setInstanceId(ValueProvider<String> instanceId);

@Default.String("my-test")
@Description("The Cloud Bigtable table ID in the instance." )
ValueProvider<String> getTableId();
void setTableId(ValueProvider<String> tableId);

@Description("bucket name")
@Default.String("mybucket")
ValueProvider<String> getBucketName();
void setBucketName(ValueProvider<String> bucketName);

任何帮助将不胜感激.

推荐答案

我确实认为在构造时验证运行时参数是一个问题.但是，我不理解的是不遵守使用模板执行管道时传递的运行时参数.

I do believe that validating runtime parameters at construction time is an issue. However, what I don't understand is not honoring the runtime parameters that were passed when executing the pipeline using the template.

如何传递运行时参数? 应该是这样的:

How do you pass your runtime parameters? It should be something like this:

  public interface WordCountOptions extends PipelineOptions {
    @Description("Path of the file to read from")
    @Default.String("gs://dataflow-samples/shakespeare/kinglear.txt")
    ValueProvider<String> getInputFile();
    void setInputFile(ValueProvider<String> value);
  }

    public static void main(String[] args) {
        WordCountOptions options =
              PipelineOptionsFactory.fromArgs(args).withValidation()
                .as(WordCountOptions.class);
        Pipeline p = Pipeline.create(options);

有关详细信息，请参见创建模板": https://cloud.google .com/dataflow/docs/templates/creating-templates

See "create template" for details: https://cloud.google.com/dataflow/docs/templates/creating-templates

一旦构建了模板，就可以使用运行时参数执行管道.例如:

Once the template is constructed, you can execute the pipeline with runtime parameters. For example:

gcloud beta dataflow jobs run test-run1 \
        --gcs-location gs://my_template/templates/DemoTemplate \
        --parameters inputFile=/path/to/my-file

有关详细信息，请参见执行模板": https://cloud.google .com/dataflow/docs/templates/executing-templates

See "Execute templates" for details: https://cloud.google.com/dataflow/docs/templates/executing-templates

注意:如果在执行管道时未传递运行时参数，则这些参数将具有默认值或null.

Note: If you don't pass runtime parameters when executing your pipeline, the parameters will either have default values or null.

希望这会有所帮助！

这篇关于在模板执行时未接受ValueProvider类型的参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！