在spark中使用hive数据库 [英] Using hive database in spark

问题描述

我是 spark 新手，正在尝试使用 HortonWorks Sandbox 在 tpcds 基准表上运行一些查询.http://www.tpc.org/tpcds/在沙箱上通过 shell 或 hive-view 使用 hive 时没有问题.问题是如果我想使用spark，我不知道如何连接到数据库.如何在 spark 中使用 hive 数据库来运行查询?到目前为止，我知道的唯一解决方案是使用以下 Scala 代码手动重建每个表并在其中加载数据，这不是最佳解决方案.

I am new in spark and trying to run some queries on tpcds benchmark tables, using HortonWorks Sandbox. http://www.tpc.org/tpcds/ There is no problem while using hive through shell or hive-view on sandbox. The problem is that I don't know how connect to the database if I want to use the spark. How can I use a hive database in spark for running the queries? The only solution that I know till now is to rebuild each table manually and load data in them using the following scala codes, which is not the best solution.

scala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
scala> sqlContext.sql("CREATE TABLE IF NOT EXISTS employee(id INT, name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'")
scala> sqlContext.sql("LOAD DATA LOCAL INPATH 'employee.txt' INTO TABLE employee")
scala> val result = sqlContext.sql("FROM employe SELECT id, name, age")
scala> result.show()

我也阅读了一些关于 hive-site.xml 的内容，但我不知道在哪里可以找到它以及对其进行哪些更改以连接到数据库.

I also read some about hive-site.xml but I don't know where to find it and what changes to make on it to connect to the database.

推荐答案

使用 Spark 和 HiveContext 时无需连接特定数据库.

There is no need to connect to a specific database when using Spark and HiveContext.

您只需将hive-site.xml"文件复制到 Spark conf 文件夹(或者您也可以创建一个符号链接).

You simply need to copy the "hive-site.xml" file to the Spark conf folder (or you could also create a symlink).

cp $HIVE_HOME/conf/hive-site.xml $SPARK_HOME/conf/

然后，在 Spark 中，您可以执行类似的操作(我不是 Scala 用户，因此语法可能有误):

Then, in Spark you can do something like that (I'm not a scala user so the syntax might be wrong) :

val hc = new org.apache.spark.sql.hive.HiveContext(sc)
val result = hc.sql("SELECT col1, col2, col3 FROM dbname.tablename")
result.show()

这篇关于在spark中使用hive数据库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！