4040和8080上的Web UI有什么区别? [英] What is the difference between web UIs on 4040 and 8080?

问题描述

有两种不同的Web UI(一种仅用于独立模式).当我以独立模式启动Spark时，可以在端口4040上使用Web UI吗? (例如:spark-class.cmd org.apache.spark.deploy.master.Master-web ui 8080在运行，4040-在不行.)这些UI之间的主要区别是什么?

There are two different web UIs (one is for standalone mode only). Can I use web UI on port 4040 when I am launching Spark in standalone mode? (example:spark-class.cmd org.apache.spark.deploy.master.Master- web ui 8080 is working, 4040 - not.) What is the main difference between these UIs?

我是否可以启动Spark(没有hadoop，hdfs，yarn等)，保持它的状态并将我的jar(类)提交给它?我想看完工作统计.我正在尝试这样的事情:

Is it possible for me to launch Spark (without hadoop, hdfs, yarn etc), to keep it up and to submit my jars(classes) into it? I want to watch job statistics after it finishes. I am trying something like this:

Server: Spark\bin>spark-class.cmd org.apache.spark.deploy.master.Master
Worker: Spark\bin>spark-class.cmd org.apache.spark.deploy.worker.Worker spark://169.254.8.45:7077 --cores 4 --memory 512M
Submit: Spark\bin>spark-submit.cmd --class demo.TreesSample --master spark://169.254.8.45:7077 file:///E:/spark-demo/target/demo.jar

它运行.为此，它将在端口4040上获取新的WebUI.我在8080的师父界面中看不到任何东西.

It runs. It gets new WebUI on port 4040 up for this task. I dont see anything in Master's ui on 8080.

当前，我正在使用win7 x64，spark-1.5.2-bin-hadoop2.6.如果有关系，我可以切换到linux.

Currently I'm using win7 x64, spark-1.5.2-bin-hadoop2.6. I can switch into linux if it matters.

推荐答案

如

You should be able to change the web UI port for standalone Master using spark.master.ui.port or SPARK_MASTER_WEBUI_PORT as described in Configuring Ports for Network Security / Standalone mode only.

Standalone Master的Web UI是集群管理器的管理控制台(它恰好是Apache Spark的一部分，但可能像Hadoop YARN和Apache Mesos一样是单独的产品).话虽如此，这通常会使两个Web UI的共同之处混淆，而答案却无济于事.

Standalone Master's web UI is a management console of a cluster manager (that happens to be part of Apache Spark, but could've been a separate product as Hadoop YARN and Apache Mesos). Having said that, it can often be confusing what the two web UIs have in common, and the answer is nothing.

Spark驱动程序的Web UI用来显示您的计算进度(作业，阶段，用于RDD持久性的存储，广播，累加器)，而独立的Master的Web UI可以让您知道操作环境"的当前状态( aka Spark独立集群).

The Spark driver's web UI is to show the progress of your computations (jobs, stages, storage for RDD persistence, broadcasts, accumulators) while standalone Master's web UI is to let you know the current state of your "operating environment" (aka the Spark Standalone cluster).

我将您有关历史记录服务器的问题的另一部分留给 @Sumit的答案.

I leave the other part of your question about History server to @Sumit's answer.

这篇关于4040和8080上的Web UI有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！