如何从Spark Shell摆脱derby.log，metastore_db [英] How to get rid of derby.log, metastore_db from Spark Shell

问题描述

运行spark-shell时，它将创建文件derby.log和文件夹metastore_db.如何配置spark将其放置在其他位置?

When running spark-shell it creates a file derby.log and a folder metastore_db. How do I configure spark to put these somewhere else?

对于derby日志，我已经尝试摆脱derby.log 一样spark-shell --driver-memory 10g --conf "-spark.driver.extraJavaOptions=Dderby.stream.info.file=/dev/null"具有几个不同的属性，但spark会忽略它们.

For derby log I've tried Getting rid of derby.log like so spark-shell --driver-memory 10g --conf "-spark.driver.extraJavaOptions=Dderby.stream.info.file=/dev/null" with a couple of different properties but spark ignores them.

有人知道如何摆脱它们或为它们指定默认目录吗?

Does anyone know how to get rid of these or specify a default directory for them?

推荐答案

自Spark 2.0.0起，不再使用hive.metastore.warehouse.dir， 参阅文档.

The use of the hive.metastore.warehouse.dir is deprecated since Spark 2.0.0, see the docs.

此答案所暗示的是，metastore_db目录和derby.log文件的真正罪魁祸首是在每个工作子目录中创建的是derby.system.home属性，默认为..

As hinted by this answer, the real culprit for both the metastore_db directory and the derby.log file being created in every working subdirectory is the derby.system.home property defaulting to ..

因此，可以通过将以下行添加到spark-defaults.conf来指定两者的默认位置:

Thus, a default location for both can be specified by adding the following line to spark-defaults.conf:

spark.driver.extraJavaOptions -Dderby.system.home=/tmp/derby

其中/tmp/derby可以替换为您选择的目录.

where /tmp/derby can be replaced by the directory of your choice.

这篇关于如何从Spark Shell摆脱derby.log，metastore_db的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！