如何访问Spark Web UI? [英] How to access Spark Web UI?

问题描述

我正在本地运行4个节点的Spark应用程序.当我运行我的应用程序时，它会显示具有该地址10.0.2.15的驱动程序:

I'm running a Spark application locally of 4 nodes. when I'm running my Application it displays my driver having this address 10.0.2.15:

INFO Utils: Successfully started service 'SparkUI' on port 4040.
INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.2.15:4040

在运行结束时显示:

INFO SparkUI: Stopped Spark web UI at http://10.0.2.15:4040
INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
INFO MemoryStore: MemoryStore cleared
INFO BlockManager: BlockManager stopped
INFO BlockManagerMaster: BlockManagerMaster stopped
INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
INFO SparkContext: Successfully stopped SparkContext

我尝试通过以下方式访问Spark Web: 10.0.2.15:4040，但是无法访问该页面. 尝试使用以下地址也无济于事:

I tried to access the Spark Web by: 10.0.2.15:4040 but the page is inaccessible. Trying with the below address also didn't helped:

 http://localhost:18080

使用ping 10.0.2.15的结果是:

Send a request 'Ping' 10.0.2.15 with 32 bytes of data

Waiting time exceeded

Waiting time exceeded

Waiting time exceeded

Waiting time exceeded

Ping statistics for 10.0.2.15: Packages: sent = 4, received = 0, lost = 4 (100% loss)

使用netstat -a检查端口4040的可用性，以验证哪些端口可用.结果如下:

Checked the availability of the port 4040 using netstat -a to verify which ports are available. The result is as follow:

   Active connexion:

    Active       local address        Remote address                      state

    TCP          127.0.0.1:4040      DESKTOP-FF4U.....:0                 Listening

PS .:知道我的代码正在成功运行. 可能是什么原因?

PS.: Knowning that my code is running succesfully. What could be the reason?

推荐答案

INFO Utils: Successfully started service 'SparkUI' on port 4040.
INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.2.15:4040

这就是Spark报告Web UI(内部称为SparkUI)绑定到端口4040的方式.

That's how Spark reports that the web UI (which is known as SparkUI internally) is bound to the port 4040.

只要Spark应用程序已启动并正在运行，您就可以通过 http://10.0.2.15访问Web UI: 4040 .

As long as the Spark application is up and running, you can access the web UI at http://10.0.2.15:4040.

INFO SparkUI: Stopped Spark web UI at http://10.0.2.15:4040
...
INFO SparkContext: Successfully stopped SparkContext

这是一个Spark应用程序完成时(它是否正确完成并不重要).从现在开始，Web UI(位于 http://10.0.2.15:4040 )不再可用.

This is when a Spark application has finished (it does not really matter whether it finished properly or not). From now on, the web UI (at http://10.0.2.15:4040) is no longer available.

我尝试通过以下方式访问Spark Web:10.0.2.15:4040，但该页面无法访问.

I tried to access the Spark Web by: 10.0.2.15:4040 but the page is inaccessible.

这是Spark应用程序的预期行为.完成后，4040(这是Web UI的默认端口)将不再可用.

That's the expected behaviour of a Spark application. Once it's completed, 4040 (which is the default port of a web UI) is no longer available.

尝试使用以下地址也无济于事: http://localhost:18080

18080是Spark History Server的默认端口.这是一个单独的过程，无论运行的Spark应用程序是否可用，它都可能会或可能不会.

18080 is the default port of Spark History Server. It is a separate process and may or may not be available regardless of availability of running Spark applications.

Spark History Server与Spark应用程序完全不同.引用官方Spark文档:

Spark History Server is completely different from a Spark application. Quoting the official Spark docs:

如果存在应用程序的事件日志，仍然可以通过Spark的历史记录服务器构建应用程序的UI.您可以通过执行以下命令来启动历史记录服务器:

It is still possible to construct the UI of an application through Spark’s history server, provided that the application’s event logs exist. You can start the history server by executing:

./sbin/start-history-server.sh

默认情况下，这会在http://:18080创建一个Web界面，列出未完成和已完成的应用程序和尝试.

This creates a web interface at http://:18080 by default, listing incomplete and completed applications and attempts.

如您所见，您必须自己启动Spark History Server才能使用18080.

As you could read, you have to start Spark History Server yourself to have 18080 available.

此外，您必须使用spark.eventLog.enabled和spark.eventLog.dir配置属性，才能在Spark应用程序完成执行后查看日志.引用 Spark官方文档:

Moreover, you have to use spark.eventLog.enabled and spark.eventLog.dir configuration properties to be able to view the logs of Spark applications once they're completed their execution. Quoting the Spark official docs:

spark作业本身必须配置为记录事件，并将它们记录到相同的共享可写目录中.例如，如果服务器配置了日志目录hdfs://namenode/shared/spark-logs，则客户端选项将是:

The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:

spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs

这篇关于如何访问Spark Web UI?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！