如何在 Beam 2.0 中的复合 PTransform 中获取 PipelineOptions? [英] How to get PipelineOptions in composite PTransform in Beam 2.0?

问题描述

升级到 Beam 2.0 后，Pipeline 类不再有 getOptions() 类.我有一个复合 PTransform，它依赖于在其 expand 方法中获取选项:

After upgrading to Beam 2.0 the Pipeline class doesn't have getOptions() class anymore. I have a composite PTransform that relies on getting the options in the its expand method:

public class MyCompositeTransform extends PTransform<PBegin, PDone> {
    @Override
    public PDone expand(PBegin input) {
        Pipeline pipeline = input.getPipeline();
        MyPipelineOptions options = pipeline.getOptions().as(MyPipelineOptions.class);
        ...
    }
}

在 Beam 2.0 中，似乎没有办法在 expand 方法中访问 PipelineOptions.

In Beam 2.0 there doesn't seem to be a way to access the PipelineOptions at all in the expand method.

有什么选择?

推荐答案

Pablo 的回答是正确的.我还想澄清一下，PipelineOptions 的管理方式发生了重大变化.

Pablo's answer is right on. I want to also clarify that there is a major change in how PipelineOptions are managed.

您可以使用它们来解析参数并将其传递给您的 main 程序(或任何构建管道的代码)，但这些在技术上独立于配置的 PipelineOptions您的管道是如何运行的.

You can use them to parse and pass around the arguments to your main program (or whatever code builds your pipeline) but these are technically independent from the PipelineOptions that configure how your pipeline is run.

在 Beam 中，Pipeline 是完全构建的，只有在此之后您才能选择一个 PipelineRunner 和 PipelineOptions 来控制管道的运行方式.管道本身实际上没有选项.

In Beam, the Pipeline is fully constructed, and only afterwards you choose a PipelineRunner and PipelineOptions to control how the pipeline is run. The pipeline itself does not actually have options.

如果您确实希望 PTransform(而不是它的扩展)的行为使用一些动态获取的选项，您应该让您的 PTransform 接受一个 ValueProvider 像WriteFiles 中的这个例子，你可以定义一个管道选项，它返回一个ValueProvider 喜欢这里 ValueProviderTest

If you do want the behavior of your PTransform (not its expansion) to use some option that is obtained dynamically, you should make your PTransform accept a ValueProvider like this example in WriteFiles and you can define a pipeline option that returns a ValueProvider like here in ValueProviderTest

这篇关于如何在 Beam 2.0 中的复合 PTransform 中获取 PipelineOptions?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！