Google Dataproc上的Spark UI在哪里? [英] Where is the Spark UI on Google Dataproc?

问题描述

我应该使用哪个端口访问Google Dataproc上的Spark UI?

What port should I use to access the Spark UI on Google Dataproc?

我尝试使用端口4040和7077以及使用netstat -pln

I tried port 4040 and 7077 as well as a bunch of other ports I found using netstat -pln

防火墙已正确配置.

推荐答案

Dataproc在YARN上运行Spark，因此您将找不到典型的"Spark独立"端口.相反，在运行Spark作业时，您可以访问端口8088，该端口将显示YARN ResourceManager的主页.任何正在运行的Spark作业都可以通过该页面上的Application Master链接进行访问.对于默认的Spark设置，Spark Application Master的页面看起来与通常在端口8080上常见的熟悉的Spark-standalone登陆页面相同.

Dataproc runs Spark on top of YARN, so you won't find the typical "Spark standalone" ports; instead, when running a Spark job, you can visit port 8088 which will show you the YARN ResourceManager's main page. Any running Spark jobs will be accessible through the Application Master link on that page. The Spark Application Master's page looks the same as the familiar Spark-standalone landing page that you would normally find on port 8080 for default Spark setups.

由于工作人员通过内部网络签入，YARN的链接将使用群集内部的主机名(主机名应包括您的Dataproc群集名称作为前缀)，但这意味着，如果您是从外部网络访问，则链接一开始可能无法工作；如果您使用的是基于防火墙的方法，则必须用外部IP地址替换主机名.

Since workers check in over the internal network, YARN's links will be using cluster-internal hostnames (the hostnames should include your Dataproc cluster name as a prefix), but this means if you're accessing from the outside network, the links may not work at first; you have to replace the hostname with the external IP address if you're using the firewall-based approach.

一种更简单的体验将是使用SOCKS代理方法，如下所述: https://cloud.google. com/dataproc/cluster-web-interfaces

An easier experience will be to use the SOCKS proxy approach as explained here: https://cloud.google.com/dataproc/cluster-web-interfaces

在这种情况下，只需使用gcloud compute ssh运行轻量级本地袜子代理，然后打开指向该浏览器的浏览器，即可正常单击所有YARN链接.

In that case, simply using gcloud compute ssh to run a lightweight local socks proxy and then opening a browser pointed at that will let you click all the YARN links as normal.

这篇关于Google Dataproc上的Spark UI在哪里?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！