无法使用mongo-hadoop连接器将Hive与MongoDB连接 [英] Unable to connect Hive with MongoDB using mongo-hadoop connector

问题描述

我试图用mongo-hadoop-core 2.0.2首次安装和配置 hive 。我已经安装了 hadoop 2.8.0，Hive 2.1.1和MongoDB 3.4.6。，当单独运行时一切正常。

我的问题是的，我无法将MongoDB与Hive连接起来。我正在使用mongo-Hadoop连接器，如 https：// github。 com / mongodb / mongo-hadoop / wiki / Hive-Usage
所需的jar添加到Hadoop和Hive lib中。即使我将它们从hive控制台添加到hive.sh或运行时。
执行时出现错误创建表查询

My Query is

  CREATE EXTERNAL TABLE testHive 
（
 ID STRING，
名称STRING 
） 
存放于'com.mongodb.hadoop.hive.MongoStorageHandler'
与SERDEPROPERTIES（'mongo.columns.mapping'='{id：_ id，name：name} '）
 TBLPROPERTIES（'mongo.uri'='mongodb：// localhost：27017 / hiveDb.testHive'）; 
  

我得到以下错误：

  FAILED：执行错误，从org.apache.hadoop.hive.ql.exec.DDLTask返回代码1。 com / mongodb / hadoop / io / BSONWritable 
 hive>错误hive.ql.exec.DDLTask  -  java.lang.NoClassDefFoundError：com / mongodb / hadoop / io / BSONWritable 
 at com.mongodb.hadoop.hive.BSONSerDe.initialize（BSONSerDe.java:132）
 at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe（SerDeUtils.java:537）
 at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer（MetaStoreUtils.java:424）
在org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer（MetaStoreUtils.java:411）
 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore（Table.java:279）
 at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer（Table.java:261）
  

它显示com / mongodb / hadoop / io / BSONWritable类不在类路径中，但是我添加了需要的（mongo-hadoop-core.jar）jar和class。
我使用的jar版本

mongo-hadoop-core 2.0.2，

mongo-hadoop-hive 2.0.2，

mongo-java-driver 3.0.2

感谢 p>

解决方案

您需要明确注册jar。在Hive脚本中，使用 ADD JAR 命令来包含这些JAR（核心，配置单元和Java驱动程序），例如 ADD JAR / path- to / mongo-hadoop-hive-< version> .jar; 。

如果您是从Hive shell运行，请像这样使用。

hive> ADD JAR /path-to/mongo-hadoop-hive-<version>.jar;

然后执行您的查询。

I am trying to install and configure hive with mongo-hadoop-core 2.0.2, for the first time. I have installed hadoop 2.8.0, Hive 2.1.1 and MongoDB 3.4.6. and everything works fine when running individually.

My problem is, I am not able to connect MongoDB with Hive. I am using mongo-Hadoop connector for this as mentioned here https://github.com/mongodb/mongo-hadoop/wiki/Hive-Usage The required jars are added to Hadoop and Hive lib. Even I add them in hive.sh or runtime from hive console. I am getting error while executing Create table query

My Query is

CREATE EXTERNAL TABLE testHive
(
    id STRING,
    name STRING
)
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id","name":"name"}')
TBLPROPERTIES('mongo.uri'='mongodb://localhost:27017/hiveDb.testHive');

And I get the following error

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. com/mongodb/hadoop/io/BSONWritable
hive> ERROR hive.ql.exec.DDLTask - java.lang.NoClassDefFoundError: com/mongodb/hadoop/io/BSONWritable
    at com.mongodb.hadoop.hive.BSONSerDe.initialize(BSONSerDe.java:132)
    at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:537)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261)

It shows that com/mongodb/hadoop/io/BSONWritable class is not in classpath but I have added the required(mongo-hadoop-core.jar) jar and class are present in the jar. The version of jars I am using

mongo-hadoop-core 2.0.2,

mongo-hadoop-hive 2.0.2,

mongo-java-driver 3.0.2

Thanks

解决方案

You need to register jars explicitly. In your Hive script, use ADD JAR commands to include these JARs (core, hive, and the Java driver), e.g., ADD JAR /path-to/mongo-hadoop-hive-<version>.jar;.

If you are running from Hive shell, use like this.

hive> ADD JAR /path-to/mongo-hadoop-hive-<version>.jar;

Then execute your query.

这篇关于无法使用mongo-hadoop连接器将Hive与MongoDB连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！