Apache Spark:如何更改Spark驱动程序侦听的端口 [英] Apache Spark : How to change the port the Spark driver listens to

问题描述

我正在尝试以Yarn作为主服务器在客户端模式下运行Spark作业，而我遇到了一个特殊问题.日志显示 Yarn ApplicationMaster正在等待Spark驱动程序可用，然后在一段时间后引发异常，指出 Yarn ApplicationMaster无法在10.60.18.77:34332连接到驱动程序.现在，我知道该IP是可访问的，但该端口不是因为防火墙阻止了它.试图让管理员打开端口很乏味.那么，有什么办法可以使Yarn Application侦听另一个端口20002呢(我知道它是开放端口，而不是34332)?

I am trying to run a Spark job in client mode with Yarn as the master and I am facing a particular problem. The log says that the Yarn ApplicationMaster is waiting for the Spark driver to be reachable and then after sometime throws an exception saying that the Yarn ApplicationMaster failed to connect to the driver at 10.60.18.77:34332. Now, I know that the IP is reachable but the port is not because the firewall is blocking it. Trying to get the admins to open a port is tedious. So is there a way I can make the Yarn Application listen to another port say 20002 which I know is open as opposed to 34332?

推荐答案

您可以通过设置 spark.driver.port = 20002

例如，在提交火花时，您可以使用类似lik的东西:

For example in your spark-submit you could use something lik:

$ spark-submit --master yarn-client --conf spark.driver.port = 20002 ....

这篇关于Apache Spark:如何更改Spark驱动程序侦听的端口的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！