spark.eventLog.dir和spark.history.fs.logDirectory有什么区别? [英] What's the difference between spark.eventLog.dir and spark.history.fs.logDirectory?

问题描述

在Spark中，事件日志目录和历史服务器日志目录之间有什么区别?

In Spark, what is the difference between the event log directory and the history server log directory?

spark.eventLog.dir hdfs:///var/log/spark/apps
spark.history.fs.logDirectory hdfs:///var/log/spark/apps

推荐答案

摘自Apache Spark的官方文档:

From the official documentation of Apache Spark:

spark.eventLog.dir是Spark事件记录在其中的基本目录，如果spark.eventLog.enabled为true.在此基本目录中，Spark为每个应用程序创建一个子目录，并将特定于该应用程序的事件记录在此目录中.用户可能希望将其设置为HDFS目录之类的统一位置，以便历史记录服务器可以读取历史记录文件.

spark.eventLog.dir is the base directory in which Spark events are logged, if spark.eventLog.enabled is true. Within this base directory, Spark creates a sub-directory for each application, and logs the events specific to the application in this directory. Users may want to set this to a unified location like an HDFS directory so history files can be read by the history server.

请参见 spark.eventLog.dir .

spark.history.fs.logDirectory用于文件系统历史记录提供程序，该目录的URL包含要加载的应用程序事件日志.这可以是本地file://路径，HDFS路径hdfs://namenode/shared/spark-logs或Hadoop API支持的备用文件系统的路径.

spark.history.fs.logDirectory is for the filesystem history provider, the URL to the directory containing application event logs to load. This can be a local file:// path, an HDFS path hdfs://namenode/shared/spark-logs or that of an alternative filesystem supported by the Hadoop APIs.

请参见 spark.history.fs.logDirectory

spark.eventLog.dir用于生成日志，而spark.history.fs.logDirectory是Spark History Server查找日志事件的位置.

spark.eventLog.dir is to generate logs while spark.history.fs.logDirectory is the place where Spark History Server finds log events.

这篇关于spark.eventLog.dir和spark.history.fs.logDirectory有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！