ValueProvider 类型参数在模板执行时没有得到遵守 [英] ValueProvider type parameters not getting honored at the template execution time

问题描述

我试图在执行时传递在 TemplateOption 类中定义为 ValueProvider 的 BigTable tableId、instanceId 和 projectId，因为它们是运行时值，但它们没有得到新值的尊重.管道使用构建管道时定义的旧值执行.我应该进行哪些更改才能在运行时尊重值?

I am trying to pass BigTable tableId, instanceId and projectId which are defined as ValueProvider in the TemplateOption class at the execution time as they are runtime values but they don't get honored with the new values . The pipleine gets executed with the old values which were defined when the pipeline was constructed. What changes should i make so that it honors values at runtime?

Pipeline p = Pipeline.create(options);
com.google.cloud.bigtable.config.BigtableOptions.Builder optionsBuilder =
        new com.google.cloud.bigtable.config.BigtableOptions.Builder().
                setProjectId("my-project");   

PCollection<com.google.bigtable.v2.Row> row = p.apply("filtered read", org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.read().withBigtableOptions(optionsBuilder).withoutValidation().withInstanceId(options.getInstanceId()).withProjectId(options.getProjectId()).withTableId(options.getTableId()));
PCollection<KV<Integer,String>> convertToKV = row.apply(ParDo.of(new ConvertToKV()));  

我的选项类看起来像:--

My Option class looks like :--

@Default.String("my-project")
@Description("The Google Cloud project ID for the Cloud Bigtable instance.")
ValueProvider<String> getProjectId();
void setProjectId(ValueProvider<String> projectId);

@Default.String("my-instance")
@Description("The Google Cloud Bigtable instance ID .")
ValueProvider<String> getInstanceId();
void setInstanceId(ValueProvider<String> instanceId);

@Default.String("my-test")
@Description("The Cloud Bigtable table ID in the instance." )
ValueProvider<String> getTableId();
void setTableId(ValueProvider<String> tableId);

@Description("bucket name")
@Default.String("mybucket")
ValueProvider<String> getBucketName();
void setBucketName(ValueProvider<String> bucketName);

任何帮助将不胜感激.

推荐答案

我确实认为在构建时验证运行时参数是一个问题.但是，我不明白的是不尊重使用模板执行管道时传递的运行时参数.

I do believe that validating runtime parameters at construction time is an issue. However, what I don't understand is not honoring the runtime parameters that were passed when executing the pipeline using the template.

你如何传递你的运行时参数?应该是这样的:

How do you pass your runtime parameters? It should be something like this:

  public interface WordCountOptions extends PipelineOptions {
    @Description("Path of the file to read from")
    @Default.String("gs://dataflow-samples/shakespeare/kinglear.txt")
    ValueProvider<String> getInputFile();
    void setInputFile(ValueProvider<String> value);
  }

    public static void main(String[] args) {
        WordCountOptions options =
              PipelineOptionsFactory.fromArgs(args).withValidation()
                .as(WordCountOptions.class);
        Pipeline p = Pipeline.create(options);

有关详细信息，请参阅创建模板":https://cloud.google.com/dataflow/docs/templates/creating-templates

See "create template" for details: https://cloud.google.com/dataflow/docs/templates/creating-templates

构建模板后，您可以使用运行时参数执行管道.例如:

Once the template is constructed, you can execute the pipeline with runtime parameters. For example:

gcloud beta dataflow jobs run test-run1 \
        --gcs-location gs://my_template/templates/DemoTemplate \
        --parameters inputFile=/path/to/my-file

有关详细信息，请参阅执行模板":https://cloud.google.com/dataflow/docs/templates/executing-templates

See "Execute templates" for details: https://cloud.google.com/dataflow/docs/templates/executing-templates

注意:如果您在执行管道时不传递运行时参数，则参数将具有默认值或空值.

Note: If you don't pass runtime parameters when executing your pipeline, the parameters will either have default values or null.

希望这会有所帮助！

这篇关于ValueProvider 类型参数在模板执行时没有得到遵守的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！