Apache Spark无法创建临时目录 [英] Apache Spark Failed to create Temp Directory

问题描述

我是Apache-Spark的新手，尝试从我的应用程序(tomcat)创建SparkConf，我注意到以下错误

I am a newbie to Apache-Spark and trying to create SparkConf from my application (tomcat) and I notice the below error

ERROR [RMI TCP Connection(4)-127.0.0.1] DiskBlockManager.logError(91) | Failed to create local dir in /opt/software/apache-tomcat-7.0.70/temp. Ignoring this directory.
java.io.IOException: Failed to create a temp directory (under /opt/software/apache-tomcat-7.0.70/temp) after 10 attempts!
    at org.apache.spark.util.Utils$.createDirectory(Utils.scala:285)

我尚未在spark-env.sh中配置 SPARK_LOCAL_DIRS .默认情况下已将其注释掉.

I haven't configured SPARK_LOCAL_DIRS in spark-env.sh. It's commented out as per default installation.

我认为我的问题与 apache火花有关，未能创建任何本地目录" ，但不是很确定.是否应将SPARK_LOCAL_DIRS配置为每个主节点/工作节点中的有效目录?如果是这样，该目录必须具有的最小磁盘空间是多少?

I think my issue is related to apache spark, "failed to create any local dir" but not very sure. Should I configure SPARK_LOCAL_DIRS to a valid directory in each of master/worker nodes ? If so what's the minimum disk space that directory must have ?

推荐答案

• 我尚未在spark-env.sh中配置SPARK_LOCAL_DIRS.评论了默认安装.

  如果您未配置，则它将采用默认位置/temp

  If you don't configure,then it would take default location /temp

  我应该将SPARK_LOCAL_DIRS配置为每个目录中的有效目录主节点/工作节点?是的.最好将其配置为您的应用程序可能需要一定数量空间的位置.

  Should I configure SPARK_LOCAL_DIRS to a valid directory in each of master/worker nodes ? Yes.its better to configure to a location where you have decent amount space that your application might require.

  如果是，该目录必须具有的最小磁盘空间是多少?

  取决于您的应用程序大小，预期要运行的作业数，每个工作程序生成的日志量，要处理的数据量(中间改组数据将存储在此处)，缓存的数量和数据量将要缓存等等.我在应用程序中配置了100 + GB的空间，它肯定取决于我刚才提到的参数.

  Depends on your application size, number of jobs you expected to be run, amount of logs each worker generates, amount of data you going to handle (intermediate shuffling data would be stored here), number of cache and volume of data you are about to cache and so on. I'm configuring decent 100+GB space in my application and its certainly depends on the parameters just I mentioned.

  这篇关于Apache Spark无法创建临时目录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！