Apache的火花，＆QUOT;未能创造任何本地目录＆QUOT; [英] apache spark, "failed to create any local dir"

问题描述

我试图安装Apache的星火一个独立的小簇（1主节点和8从属节点）上。我已经安装了pre-建建立在Hadoop的2.4之上火花1.1.0版本。我已经建立了节点间的无密码的ssh和出口一些必要的环境变量。其中一个变量（这可能是最相关的）是：

I am trying to setup Apache-Spark on a small standalone cluster (1 Master Node and 8 Slave Nodes). I have installed the "pre-built" version of spark 1.1.0 built on top of Hadoop 2.4. I have set up the passwordless ssh between nodes and exported a few necessary environment variables. One of these variables (which is probably most relevant) is:

export SPARK_LOCAL_DIRS=/scratch/spark/

我有一小片蟒蛇code的，我知道有星火工程。我的桌面上，而不是集群 - - 我可以在本地运行它：

I have a small piece of python code which I know works with Spark. I can run it locally--on my desktop, not the cluster--with:

$SPARK_HOME/bin/spark-submit ~/My_code.py

我复制了code到集群。然后，我开始的所有进程从头节点：

I copied the code to the cluster. Then, I start all the processes from the head node:

$SPARK_HOME/sbin/start-all

和每一个奴隶被列为运行的过程XXXXX。

And each of the slaves is listed as running as process xxxxx.

如果我再尝试与上述相同的命令来运行我的code：

If I then attempt to run my code with the same command above:

$SPARK_HOME/bin/spark-submit ~/MY_code.py

我收到以下错误：

I get the following error:

14/10/27 14:19:02 ERROR util.Utils: Failed to create local root dir in /scratch/spark/.  Ignoring this directory.
14/10/27 14:19:02 ERROR storage.DiskBlockManager: Failed to create any local dir.

我在777上的 /划伤和 /刮/火花设置权限任何帮助是极大AP preciated。

I have the permissions set on the /scratch and /scratch/spark at 777. Any help is greatly appreciated.

推荐答案

但问题是我不知道的主节点还需要一个临时目录。在我的每个8工作节点的我创建的本地/刮/火花目录，但忽视了对主节点上这样做。添加目录解决了这一问题。

The problem was that I didn't realize the master node also needed a scratch directory. In each of my 8 worker nodes I created the local /scratch/spark directory, but neglected to do so on the master node. Adding the directory fixed the problem.

这篇关于Apache的火花，＆QUOT;未能创造任何本地目录＆QUOT;的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！