SerDe问题与Hive 0.12和Hadoop 2.2.0-cdh5.0.0-beta2 [英] SerDe problems with Hive 0.12 and Hadoop 2.2.0-cdh5.0.0-beta2

问题描述

标题有点奇怪，因为我很难缩小这个问题。我在Hadoop 2.0.0-cdh4.4.0和hive 0.10上使用了我的解决方案，没有问题。

• 我无法使用此SerDe创建表： https://github.com/rcongiu/Hive-JSON-Serde

<第一次尝试：

  FAILED：执行错误，从org.apache.hadoop.hive.ql.exec返回代码1。 DDLTask。 org.apache.hadoop.hive.serde2.objectinspector.primitive.AbstractPrimitiveJavaObjectInspector。< init>（Lorg / apache / hadoop / hive / serde2 / objectinspector / primitive / PrimitiveObjectInspectorUtils $ PrimitiveTypeEntry;）V 
  
 
 $ b 
第二次尝试：
 
 
 失败：执行错误，从org.apache.hadoop.hive.ql.exec.DDLTask返回代码1。无法初始化类org.openx.data.jsonserde.objectinspector.JsonObjectInspectorFactory 
  
 
 
 
 
 
 
我可以使用此SerDe创建表格： https://github.com/cloudera/ cdh-twitter-example  
 我使用flume的tweets创建了一个外部表。我不能做SELECT * FROM tweets;
 
 
 FAILED：RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException：失败，出现异常java.lang .ClassNotFoundException：com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException：java.lang.ClassNotFoundException：com.cloudera.hive.serde.JSONSerDe 
  
 
 
我可以做SELECT ID，text FROM tweets; 
 
我可以做一个SELECT COUNT（*）FROM tweets; 
 
 
 
我无法自我加入此表：
 
 
 
执行日志位于：/tmp/jochen.debie/jochen.debie_20140311121313_164611a9-b0d8-4e53-9bda-f9f7ac342aaf.log 
 2014-03-11 12:13:30开始启动本地任务来处理地图加入;最大内存= 257294336 
执行失败，退出状态：2 
获取错误信息 
 
 
任务失败！ 
任务ID：
 Stage-5 
 
 
 
 
 
 
提到执行日志：
  2014-03-11 12：13：30,331错误mr.MapredLocalTask​​（MapredLocalTask​​.java:executeFromChildJVM(324）） - 配置单元运行时错误：映射本地工作失败
 org.apache.hadoop.hive.ql.metadata.HiveException：使用异常失败java.lang.ClassNotFoundException：com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException：java.lang.ClassNotFoundException：com .cloudera.hive.serde.JSONSerDe 
  
有谁知道如何解决这个问题，或者至少告诉我哪里问题是？
 
 
 
编辑：这可能是一个问题，我在Hadoop 2.0.0-cdh4.4.0和配置单元0.10上构建了serde？ 
 
解决方案
从我所看到的情况来看，Hive-.11 +在加入自定义SerDe时存在一个错误。 
 
 
  https://github.com/Esri/gis -tools-for-hadoop / issues / 9  
 
 
 
您可以尝试将包含SerDe类的JAR文件复制到$ HIVE_HOME / lib。
 
 
 
（我在您的问题中看到您在连接和其他情况下都有ClassNotFoundException;到目前为止，我遇到过的这种情况都与join有关。）
 
 
 
 另一个解决方法是使用HADOOP_CLASSPATH：
 
 
 
  env HADOOP_CLASSPATH = some.jar：other.jar配置单元...  
 
 
 
 变通适用于Hive版本0.11和0.12;那么0.13及以上包含 HIVE-6670 的修正。
 
The title is a bit weird as I'm having difficulties narrowing down the problem. I used my solution on Hadoop 2.0.0-cdh4.4.0 and hive 0.10 without issues. 


I can't create a table using this SerDe: https://github.com/rcongiu/Hive-JSON-Serde


first try:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.hadoop.hive.serde2.objectinspector.primitive.AbstractPrimitiveJavaObjectInspector.<init>(Lorg/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils$PrimitiveTypeEntry;)V
second try:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Could not initialize class org.openx.data.jsonserde.objectinspector.JsonObjectInspectorFactory



I can create a table with this SerDe: https://github.com/cloudera/cdh-twitter-example
I create an external table with tweets from flume. I can't do "SELECT * FROM tweets;"

FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe
I can do SELECT id, text FROM tweets;
I can do a SELECT COUNT(*) FROM tweets;
I can't self join this table:

Execution log at: /tmp/jochen.debie/jochen.debie_20140311121313_164611a9-b0d8-4e53-9bda-f9f7ac342aaf.log
2014-03-11 12:13:30     Starting to launch local task to process map join;      maximum memory = 257294336
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-5


mentioned execution log:
2014-03-11 12:13:30,331 ERROR mr.MapredLocalTask (MapredLocalTask.java:executeFromChildJVM(324)) - Hive Runtime Error: Map local work failed
org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe
Does anyone know how to fix this or at least show me where the problem is?

EDIT: Can it be a problem that I built the serde on a Hadoop 2.0.0-cdh4.4.0 and hive 0.10?
解决方案
From what I've seen, Hive-.11+ has a bug in join with custom SerDe.

https://github.com/Esri/gis-tools-for-hadoop/issues/9

You might try the workaround of copying the JAR file containing the SerDe class, to $HIVE_HOME/lib .

(I see in your question you got ClassNotFoundException both in join and in other cases; so far the times I have encountered such were all with join.)

[Edit] Another workaround is to use HADOOP_CLASSPATH:

env HADOOP_CLASSPATH=some.jar:other.jar hive ...

[Edit] The work around applies to Hive versions 0.11 and 0.12; then 0.13 and above contain the fix for HIVE-6670.

                        
这篇关于SerDe问题与Hive 0.12和Hadoop 2.2.0-cdh5.0.0-beta2的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！