Spark - java IOException:无法在/tmp/blockmgr* 中创建本地目录 [英] Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

问题描述

我试图运行长时间运行的 Spark 作业.执行几个小时后，我得到以下异常:

I was trying to run a long running Spark Job. After few hours of execution, I get exception below :

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28 

试图通过检查来绕过它:

Tried to get around it by checking:

1. /tmp 目录中的权限问题.Spark 服务器未以 root 身份运行.但是/tmp 目录应该对所有用户都可写.

1. Permission issue in /tmp dir. The spark server is not running as root. but /tmp dir should be writable to all users.

/tmp 目录有足够的空间.

/tmp Dir has enough space.

推荐答案

假设您正在处理多个节点，则需要检查每个节点都参与了 spark 操作(master/driver+ 奴隶/节点/工人).

Assuming that you are working with several nodes, you'll need to check every node participate in the spark operation (master/driver + slaves/nodes/workers).

请确认每个worker/node都有足够的磁盘空间(尤其是检查/tmp文件夹)，以及正确的权限.

Please confirm that each worker/node have enough disk space (especially check /tmp folder), and right permissions.

这篇关于Spark - java IOException:无法在/tmp/blockmgr* 中创建本地目录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！