可以直线和火星壳为相同的apache火花显示不同的数据库吗？ [英] Can beeline and spark-shell show different databases for same apache spark?

问题描述

我在Ubuntu上安装了hadoop 2.8.1，然后在其上安装了spark-2.2.0-bin-hadoop2.7。
首先，当我通过spark-shell创建数据库并尝试通过java JDBC程序访问它时，我没有存在表格。然后我用直线观察数据库在这里也不存在。我通过直线创建了数据库。为什么spark-shell和beeline会显示不同的数据库？

它们可能应该显示相同。我尝试了连接hive2并获取表格的普通JDBC程序，并观察到有时会显示我通过spark-shell创建的表格，有时会显示通过直线创建的表格...请帮助。同样的情况也发生在直线上。

解决方案

这可能是因为您的Spark安装
配置不正确访问您的Hive仓库。

在这种情况下，Spark-SQL旨在设置自己的本地仓库
并恢复为独立模式。这是为了减少对非Hive用户的收入
。

要解决这个问题，您应该：

1. 请参阅官方文档。
   
2. 阅读日志并查找与'hive'或'metastore'有关的任何内容到
   了解会发生什么



3. 确保Spark有权访问hive-site.xml配置文件。
   你可以设置一个符号链接（一定要首先检查路径）
   
   

     ln -s / etc /hive/conf/hive-site.xml /etc/spark/conf/hive-site.xml 
     



4. 确保您的Spark安装可以访问Hive jars
   （检查 $ SPARK_HOME / lib ）



5. 确保你已经使用类似这样的方式启用Hive支持：

     SparkSession.builder.enableHiveSupport（）。getOrCreate（）
     

希望这有助于您。

I have installed hadoop 2.8.1 on ubuntu and then installed spark-2.2.0-bin-hadoop2.7 on it. Firstly when I created database through spark-shell and tryied to access it through java JDBC program I was seing no table exists. Then I used beeline and observed that the databases do not exists here too. I created databases through beeline. Why does spark-shell and beeline show different DBs?

They probably should show the same. I tried the plain JDBC program which connects hive2 and fetched the tables and observed that sometimes it show tables I creatd through spark-shell and sometimes shows that of created through beeline...Please help. The same is happening sometimes with beeline too..

解决方案

This is probably because your Spark installation is not configured properly to access your Hive warehouse.

In such case, Spark-SQL is designed to setup its own local warehouse and revert to standalone mode. This is intended to ease adoption for non-Hive users.

To troubleshoot this, you should:

1. Refer to the official documentation.
2. read the logs and look for anything related to 'hive' or 'metastore' to understand what happens

3. make sure that Spark has access to the hive-site.xml configuration file. You can for instance set up a symlink (be sure to check the paths first)

   ln -s /etc/hive/conf/hive-site.xml    /etc/spark/conf/hive-site.xml
   

4. make sure that your Spark installation has access to the Hive jars (check $SPARK_HOME/lib)

5. make sure you have enabled Hive support with something like this:

   SparkSession.builder.enableHiveSupport().getOrCreate()
   

Hope this helps.

这篇关于可以直线和火星壳为相同的apache火花显示不同的数据库吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！