在Spark Web UI中看不到完成的作业 [英] Cannot see finished job in Spark web UI

问题描述

我正在使用./bin/spark-submit运行我的Spark作业.运行正常，但是打开Spark Web UI，我看不到完整列表中的作业.

I am using ./bin/spark-submit to run my Spark job. It runs fine but opening Spark web UI, I cannot see job in completed list.

./bin/spark-submit --name "myapp" --master local --conf "spark.master=spark://fahad:7077" --class com.apptest.App ~/app-0.0.1-SNAPSHOT.jar

注意:Spark版本2.0.1，正在运行1个工作进程，在localhost:8080上的主UI都从./sbin/start-*.sh脚本运行.

Note: Spark version 2.0.1, 1 worker running, master UI at localhost:8080 both worker and master ran from ./sbin/start-*.sh scripts.

推荐答案

有两种不同的UI，即常规的Spark UI和Spark History Server.

There are two different UI's, the regular Spark UI, and the Spark History Server.

完成作业后显示作业的是历史记录服务器.

The one that shows jobs after they complete is the history server.

http://spark.apache.org/docs/latest/monitoring.html

他们在文档中解释了您需要通过运行以下内容来启动它:

They explain in the docs that you need to start it by running:

./sbin/start-history-server.sh

默认情况下，这会在 http://server-url:18080 处创建一个Web界面， 列出未完成的申请和尝试.

This creates a web interface at http://server-url:18080 by default, listing incomplete and completed applications and attempts.

使用文件系统提供程序类时(请参见spark.history.provider 下面)，则必须在 spark.history.fs.logDirectory配置选项，并且应包含 每个代表应用程序事件日志的子目录.

When using the file-system provider class (see spark.history.provider below), the base logging directory must be supplied in the spark.history.fs.logDirectory configuration option, and should contain sub-directories that each represents an application’s event logs.

spark作业本身必须配置为记录事件并记录 它们到相同的共享可写目录.例如，如果 服务器配置了一个日志目录 hdfs://namenode/shared/spark-logs，那么客户端选项将 是:

The spark jobs themselves must be configured to log events, and to log them to the same shared, writeable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:

spark.eventLog.enabled true spark.eventLog.dir hdfs://namenode/shared/spark-logs

spark.eventLog.enabled true spark.eventLog.dir hdfs://namenode/shared/spark-logs

这篇关于在Spark Web UI中看不到完成的作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！