Spark SQL 不允许我创建表，抱怨默认的 Metastore 目录 [英] Spark SQL doesnt let me create a table, complains about default metastore directory

问题描述

在我的本地机器上安装了 Spark 1.5 spark-1.5.0-bin-hadoop2.6.跑 $ ./bin/spark-shell尝试过，按照 the doc 创建表格，得到这个:

Installed Spark 1.5 spark-1.5.0-bin-hadoop2.6 on my local machine. Ran $ ./bin/spark-shell Tried, following the doc to create a table, getting this:

> SQL context available as sqlContext.
> 
> scala> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
> STRING)"); 15/09/22 22:18:13 ERROR DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:file:/user/hive/warehouse/src is not a directory
> or unable to create one)  at
> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:720)

尝试为此传递 hive 参数，但没有奏效:

Tried passing the hive parameter for this, but didnt work:

> $  ./bin/spark-shell --conf hive.metastore.warehouse.dir=./ Warning:
> Ignoring non-spark config property: hive.metastore.warehouse.dir=./

终于尝试了 CLI 本身，但遇到了同样的问题.我在哪里更改配置单元仓库参数位置?我目前没有安装 Hadoop，也没有安装 hive.

Finally tried the CLI itself, but getting the same issue. Where do i change the hive warehouse parameter location ? I dont have Hadoop installed at the moment, nor hive.

谢谢，马特

推荐答案

hive 表的元数据存储在 Metastore 中，hive context 添加了对在 MetaStore 中查找表的支持.

Metadata of hive tables are strored in metastore, hive context adds support for finding tables in the MetaStore.

import org.apache.spark.sql.hive.HiveContext
val hiveContext = new HiveContext(sc)
val myDF = sql("select * from mytable")

你会得到 dataFrame 作为结果

You will get dataFrame as result

myDF: org.apache.spark.sql.DataFrame = [.....]

这篇关于Spark SQL 不允许我创建表，抱怨默认的 Metastore 目录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！