Spring Boot Yarn - 传递命令行参数 [英] Spring Boot Yarn - Passing Command line arguments

问题描述

我正在尝试在我的 Spring Boot Yarn 应用程序中传递命令行参数，但遇到了困难.我知道我可以在 yml 文档 spring.yarn.appmaster.launchcontext.arguments 中设置这些但如何从命令行设置?像 java -jar MyYarnApp.jar {arg0} {arg1} 并从我的 @YarnContainer 访问它?

i'm trying to pass command line arguments in my Spring Boot Yarn application and am having difficulties. i understand that i can set these in the yml document spring.yarn.appmaster.launchcontext.arguments but how can it from the command line? like java -jar MyYarnApp.jar {arg0} {arg1} and get access to it from my @YarnContainer?

我发现 @YarnProperties 映射到 spring.yarn.appmaster.launchcontext.arguments 但我想从命令行设置它们，而不是在 yml

i've discovered that @YarnProperties maps to spring.yarn.appmaster.launchcontext.arguments but i want to set them from the command line, not in the yml

推荐答案

当您找到 spring.yarn.client.launchcontext.arguments 和 spring.yarn 时，您已经非常接近了.appmaster.launchcontext.arguments.我们没有设置可以自动将所有命令行参数从客户端传递到应用程序管理员，然后应用程序管理员将它们传递到容器启动上下文中.不确定我们是否想要这样做，因为您肯定希望控制 YARN 容器启动上下文发生的情况.然后，使用客户端的用户可能会沿着食物链传递恶意参数.

You are pretty close on this when you found spring.yarn.client.launchcontext.arguments and spring.yarn.appmaster.launchcontext.arguments. We don't have settings which would automatically pass all command-line arguments from a client into an appmaster which would then pass them into a container launch context. Not sure if we even want to do that because you surely want to be on control what happens with YARN container launch context. User using a client could then potentially pass a rogue arguments along a food chain.

话虽如此，让我们看看我们可以用我们的简单单项目 YARN 应用指南做些什么.

Having said that, lets see what we can do with our Simple Single Project YARN Application Guide.

我们仍然需要使用这些启动上下文参数来定义我们的命令行参数，以基本上映射事物如何从客户端传递到应用程序管理员到容器.

We still need to use those launch context arguments to define our command line parameters to basically map how things are passed from a client into an appmaster into a container.

我在 application.yml 中添加的内容:

What I added in application.yml:

spring:
    yarn:
        client:
            launchcontext:
                arguments:
                    --my.appmaster.arg1: ${my.client.arg1:notset1}
        appmaster:
            launchcontext:
                arguments:
                    --my.container.arg1: ${my.appmaster.arg1:notset2}

修改Application类中的HelloPojo:

@YarnComponent
@Profile("container")
public static class HelloPojo {

    private static final Log log = LogFactory.getLog(HelloPojo.class);

    @Autowired
    private Configuration configuration;

    @Value("${my.container.arg1}")
    private String arg1;

    @OnContainerStart
    public void onStart() throws Exception {
        log.info("Hello from HelloPojo");
        log.info("Container arg1 value is " + arg1);
        log.info("About to list from hdfs root content");

        FsShell shell = new FsShell(configuration);
        for (FileStatus s : shell.ls(false, "/")) {
            log.info(s);
        }
        shell.close();
    }

}

注意我如何添加 arg1 并使用 @Value 将它映射到 my.container.arg1.我们可以使用 @ConfigurationProperties 或 @Value，它们是普通的 Spring 和 Spring Boot 功能，Boot 的参考文档 如何使用它们.

Notice how I added arg1 and used @Value to map it with my.container.arg1. We can either use @ConfigurationProperties or @Value which are normal Spring and Spring Boot functionalities and there's more in Boot's reference docs how to use those.

然后你可以修改AppIT单元测试:

You could then modify AppIT unit test:

ApplicationInfo info = submitApplicationAndWait(Application.class, new String[]{"--my.client.arg1=arg1value"});

并通过测试运行构建

./gradlew clean build

或者直接构建它而不运行测试:

or just build it without running test:

./gradlew clean build -x test

然后用你的 my.client.arg1 提交到一个真正的 hadoop 集群.

and then submit into a real hadoop cluster with your my.client.arg1.

java -jar build/libs/gs-yarn-basic-single-0.1.0.jar --my.client.arg1=arg1value

无论哪种方式，您都会看到 arg1value 登录到容器日志中:

Either way you see arg1value logged in container logs:

[2014-07-18 08:49:09.802] boot - 2003  INFO [main] --- ContainerLauncherRunner: Running YarnContainer with parameters [--spring.profiles.active=container,--my.container.arg1=arg1value]
[2014-07-18 08:49:09.806] boot - 2003  INFO [main] --- Application$HelloPojo: Container arg1 value is arg1value

使用格式 ${my.client.arg1:notset1} 还允许您自动定义默认值 notset1 如果 my.client.arg1 被用户省略.我们正在这里处理由 Spring Boot 编排的 Spring Application Context，因此您可以随意使用那里的所有好东西

Using format ${my.client.arg1:notset1} also allows you to automatically define a default value notset1 if my.client.arg1 is omitted by user. We're working on Spring Application Context here orchestrated by Spring Boot so all the goodies from there are in your disposal

如果您需要更精确地控制那些面向用户的参数(使用 args4j、jopt 等)，那么您需要为 client/appmaster/container order 使用单独的 code/jar 来创建自定义客户端 main 方法.所有其他 Spring YARN 入门指南几乎都使用多项目构建，所以看看那些.例如，如果您只想拥有第一个和第二个参数值，而无需在命令行上使用完整的 --my.client.arg1=arg1value.

If you need more precise control of those user facing arguments(using args4j, jopt, etc) then you'd need to have a separate code/jar for client/appmaster/container order to create a custom client main method. All the other Spring YARN getting started guides are pretty much using multi-project builds so look at those. For example if you just want to have first and second argument value without having a need to use full --my.client.arg1=arg1value on a command line.

告诉我们这是否适合您，以及您是否有任何其他想法可以让事情变得更简单.

Let us know if this works for you and if you have any other ideas to make things simpler.

这篇关于Spring Boot Yarn - 传递命令行参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！