Apache Flink - 在作业中无法识别自定义Java选项 [英] Apache Flink - custom java options are not recognized inside job

问题描述

我已将以下行添加到flink-conf.yaml：

I've added the following line to flink-conf.yaml:

env.java.opts： - Ddy.props.path = / PATH / TO / PROPS / FILE

env.java.opts: "-Ddy.props.path=/PATH/TO/PROPS/FILE"

启动jobmanager时（jobmanager.sh启动集群）我在日志中看到jvm选项确实已被识别

when starting jobmanager (jobmanager.sh start cluster) I see in logs that the jvm option is indeed recognized

2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JVM Options:
2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xms256m
2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xmx256m
2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -XX:MaxPermSize=256m
2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Ddy.props.path=/srv/dy/stream-aggregators/aggregators.conf
2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog.file=/srv/flink-1.2.0/log/flink-flink-jobmanager-0-flinkvm-master.log
2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog4j.configuration=file:/srv/flink-1.2.0/conf/log4j.properties
2017-02-20 12:19:23,536 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlogback.configurationFile=file:/srv/flink-1.2.0/conf/logback.xml

但是当我运行flink作业（flink run -d PROG.JAR）时，System.getProperty（dy.props.path）返回null（当打印系统属性时，我发现它确实缺席。）

but when I run a flink job (flink run -d PROG.JAR), System.getProperty("dy.props.path") returns null (and when printing the system properties, I see that it is indeed absent.)

问题实际上是 - 我如何设置flink-job代码中可用的系统属性？

The question really is - how do I set system properties that will be available inside of the flink-job's code?

推荐答案

这个问题与Flink的运行时架构非常相关 [1] 。

The question is very much connected with the runtime architecture of Flink [1].

我知道你在独立集群中运行你的工作。请记住， JobManager 和 TaskManager 在单独的jvm实例中运行。你必须考虑每个代码块的执行位置。

I understand you're running your job in standalone cluster. Remember that JobManager and TaskManagers run in separate jvm instances. You have to consider where will each block of code executed.

例如转换中的代码，如 map 或 filter 在 TaskManager 上执行。
您的输入类的 main 方法中的代码在命令行工具 flink 上执行，当然这当然没有设置系统属性，因为它只为作业提交产生临时（-d）jvm。

For instance code in transformations like map or filter is executed on TaskManager. Code in main method of your entry class is executed on the command line tool flink, which of course does not have the system property set as it spawns temporary(-d) jvm just for job submission.

如果您通过 WebUI 提交作业 main 方法在 JobManager 上执行，因此该属性将被设置。

If you submit your job through WebUI the code from your main method is executed on the JobManager so the property will be set then.

但基本上传递程序参数通过系统属性我会说相当沮丧。

But basically passing program arguments through system properties is I would say rather discouraged.

下面你有一个简单的例子：

Below you have a simple example:

我已经开始：

• JobManager with env.java.opts： - Ddy.props.path = jobmanager


• TaskManager with env.java.opts： - Ddy.props.path = taskmanager

• JobManager with env.java.opts:"-Ddy.props.path=jobmanager"
• TaskManager with env.java.opts:"-Ddy.props.path=taskmanager"

我的工作代码如下：

object Main {
  def main(args: Array[String]): Unit = {
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val stream = env.fromCollection(1 to 4)

    val prop = System.getProperty("dy.props.path")
    stream.map(_ => System.getProperty("dy.props.path") + "  mainArg: " + prop).print()

    env.execute("stream")
  }
}

当我通过 flink 工具提交代码时输出如下：

When I submit the code through flink tool the output is as follows:

taskmanager  mainArg: null
taskmanager  mainArg: null
taskmanager  mainArg: null
taskmanager  mainArg: null

通过 WebUI提交我得到：

taskmanager  mainArg: jobmanager
taskmanager  mainArg: jobmanager
taskmanager  mainArg: jobmanager
taskmanager  mainArg: jobmanager

这篇关于Apache Flink - 在作业中无法识别自定义Java选项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！