HDFS 上的 root 临时目录:/tmp/hive 应该是可写的.当前权限为:rwx---------(在 Linux 上) [英] The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx--------- (on Linux)
问题描述
HDFS 上的根目录:/tmp/hive 应该是可写的.当前权限为:rwx--------
以下 Spark 代码我在 CDH 5.8 & Eclipse 中执行超越运行时异常
Hi, The following Spark code i was executing in Eclipse of CDH 5.8 & getting above RuntimeExeption
public static void main(String[] args) {
final SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("HiveConnector");
final JavaSparkContext sparkContext = new JavaSparkContext(sparkConf);
SQLContext sqlContext = new HiveContext(sparkContext);
DataFrame df = sqlContext.sql("SELECT * FROM test_hive_table1");
//df.show();
df.count();
}
根据异常/tmp/hive on HDFS 应该是可写的,但是我们在本地模式下执行 spark 作业.这意味着本地 (linux) 文件系统中的目录 /tmp/hive 没有可写权限,而不是 HDFS.
According to Exception /tmp/hive on HDFS should be writable, however we are executing spark job in local mode. That means there is no writable permission to the directory /tmp/hive in local (linux) file system, not HDFS.
所以我执行了下面的命令来授予权限.
So I had executed below command to gave permission.
$ sudo chmod -R 777 /tmp/hive
现在它对我有用.
如果您在集群模式下执行 spark 作业时遇到同样的问题,您应该在 hive conf 文件夹的 hive-site.xml 文件中配置以下属性并重新启动蜂巢服务器.
If you are getting the same issue during execution of spark job in cluster mode you should configure below property in hive-site.xml file of hive conf folder and restart hive server.
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>hive.scratch.dir.permission</name>
<value>777</value>
<description>The permission for the user-specific scratch directories that get created in the root scratch directory </description>
</property>
推荐答案
使用适当的 64 位 winutils 并设置权限
use proper 64bit winutils and set permission
winutils.exe chmod -R 777 \tmp\hive
winutils.exe chmod -R 777 \tmp\hive
System.setProperty("hadoop.home.dir", "C:\\Users\\Hadoop_home")
lazy val spark: SparkSession = {
FileUtils.deleteDirectory(new File("c:\\tmp\\metastore_db"))
FileUtils.deleteDirectory(new File("c:\\tmp\\spark-warehouse"))
SparkSession.builder().config("spark.sql.warehouse.dir", "C:\\temp\\").master("local").appName("spark session for testing").enableHiveSupport().getOrCreate()
}
这篇关于HDFS 上的 root 临时目录:/tmp/hive 应该是可写的.当前权限为:rwx---------(在 Linux 上)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!