如何更改spark.ui.port? [英] How do I change the spark.ui.port?
问题描述
我正在使用spark-submit并尝试在spark上下文中使用.setExecutorEnv("spark.ui.port", "4050")
在jar文件中执行此操作,但是它仍然尝试达到4040.然后我尝试将--conf spark.ui.port=4050
放在spark-submit
之后和之前--class CLASSNAME
,但这也不起作用,这一次是说错误:无法识别的选项'--conf'".我该如何解决?我遇到的实际错误是,有一个正在使用的活动Spark服务器供其他人使用,这阻止了此Spark提交启动码头服务器.然后它没有占用其他端口,所以我试图强迫它这样做.
I am using spark-submit and tried to do this in the jar file with .setExecutorEnv("spark.ui.port", "4050")
on the spark context, but it still tried to hit 4040. I then tried to put a --conf spark.ui.port=4050
after spark-submit
and before --class CLASSNAME
, but that didn't work either, this time saying "Error: Unrecognized option '--conf'". How do I get around this? The actual error I'm running into is that there is an active spark server that others are using that is preventing this spark-submit from starting the jetty server. It's then not hitting up other ports, so I'm trying to force it to do that.
推荐答案
--conf spark.ui.port=4050
是Spark 1.1的功能.您可以在代码中进行设置,例如:
--conf spark.ui.port=4050
is a Spark 1.1 feature. You can set it in your codes, such as:
val conf = new SparkConf().setAppName(s"SimpleApp").set("spark.ui.port", "4050")
val sc = new SparkContext(conf)
这篇关于如何更改spark.ui.port?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!