在嵌入模式星火 - 用户/蜂巢/仓库未发现 [英] Spark on embedded mode - user/hive/warehouse not found

问题描述

我用在嵌入式本地模式的Apache的火花。我已经包括在我的的pom.xml 的和在同一个版本的所有依存关系（火花core_2.10，火花sql_2.10和火花hive_2.10）。

I'm using Apache Spark in embedded local mode. I have all the dependencies included in my pom.xml and in the same version (spark-core_2.10, spark-sql_2.10, and spark-hive_2.10).

我只是想运行一个查询HiveQL创建一个表（存储为实木复合地板）。

I just want to run a HiveQL query to create a table (stored as Parquet).

运行下面的（很简单）code：

Running the following (rather simple) code:

public class App {
    public static void main(String[] args) throws IOException, ClassNotFoundException {

        SparkConf sparkConf = new SparkConf().setAppName("JavaSparkSQL").setMaster("local[2]").set("spark.executor.memory", "1g");
        JavaSparkContext ctx = new JavaSparkContext(sparkConf);
        HiveContext sqlContext = new org.apache.spark.sql.hive.HiveContext(ctx.sc());

        String createQuery = "CREATE TABLE IF NOT EXISTS Test (id int, name string) STORED AS PARQUET";
        sqlContext.sql(createQuery);
    }
}

...还是返回了以下异常：

...is returning the following exception:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:file:/user/hive/warehouse/test is not a directory or unable to create one)

我可以看到在项目的根目录创建 metastore_db 文件夹中。

我搜索了一圈，发现没有帮助他们 - 最的解决方案不是为了嵌入模式。

I searched around and the solutions found didn't help --most of them were not for the embedded mode.

• 一个解决办法是检查的权限，我用的一切相同的用户。


• 另一种解决方案是在HDFS手动创建的文件夹，我没有和我可以浏览到/用户/蜂巢/仓储/测试。


• 一个解决方案是通过添加手动设置metastore： sqlContext.sql（SET hive.metastore.warehouse.dir = HDFS：//本地主机：9000 /用户/蜂巢/仓库）; 。

• One solution was to check the permissions, I'm using the same user for everything.
• Another solution was to create the folder manually in HDFS, I did and I can navigate to /user/hive/warehouse/test.
• One solution was to set manually the metastore by adding: sqlContext.sql("SET hive.metastore.warehouse.dir=hdfs://localhost:9000/user/hive/warehouse");.

我跑出来的想法，现在，有人可以提供任何其他建议？

I'm running out of ideas right now, can someone provide any other suggestions?

推荐答案

由于您是在本地嵌入式模式下运行，HDFS不被考虑。这就是为什么说，误差文件：/用户/蜂巢/仓储/测试，而不是 HDFS：//本地主机：9000 /用户/蜂巢/仓库/测试。该公司预计 /用户/蜂巢/仓储/测试到本地计算机上存在。尝试在本地创建它。

Because you're running in local embedded mode, HDFS is not being considered. This is why the error says file:/user/hive/warehouse/test rather than hdfs://localhost:9000/user/hive/warehouse/test. It expects /user/hive/warehouse/test to exist on your local machine. Try creating it locally.

这篇关于在嵌入模式星火 - 用户/蜂巢/仓库未发现的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！