Spark-Java IOException:无法在/tmp/blockmgr *中创建本地目录 [英] Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

问题描述

我试图运行长时间运行的Spark Job.执行几个小时后，我在下面看到异常:

I was trying to run a long running Spark Job. After few hours of execution, I get exception below :

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28 

尝试通过以下方法解决此问题:

Tried to get around it by checking:

1. /tmp目录中的权限问题. Spark服务器未以root用户身份运行.但是/tmp dir应该对所有用户都可写.

1. Permission issue in /tmp dir. The spark server is not running as root. but /tmp dir should be writable to all users.

/tmp目录有足够的空间.

/tmp Dir has enough space.

推荐答案

假设您正在使用多个节点，则需要检查每个节点是否参与spark操作(主设备/驱动程序+奴隶/节点/工人).

Assuming that you are working with several nodes, you'll need to check every node participate in the spark operation (master/driver + slaves/nodes/workers).

请确认每个工作服务器/节点都有足够的磁盘空间(尤其是检查/tmp文件夹)，并具有正确的权限.

Please confirm that each worker/node have enough disk space (especially check /tmp folder), and right permissions.

这篇关于Spark-Java IOException:无法在/tmp/blockmgr *中创建本地目录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！