Hive 0.12 和 Hadoop 2.2.0-cdh5.0.0-beta2 的 SerDe 问题 [英] SerDe problems with Hive 0.12 and Hadoop 2.2.0-cdh5.0.0-beta2

问题描述

标题有点奇怪，因为我很难缩小问题的范围.我在 Hadoop 2.0.0-cdh4.4.0 和 hive 0.10 上使用了我的解决方案，没有问题.

The title is a bit weird as I'm having difficulties narrowing down the problem. I used my solution on Hadoop 2.0.0-cdh4.4.0 and hive 0.10 without issues.

• 我无法使用此 SerDe 创建表:https://github.com/rcongiu/Hive-JSON-Serde

第一次尝试:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.hadoop.hive.serde2.objectinspector.primitive.AbstractPrimitiveJavaObjectInspector.<init>(Lorg/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils$PrimitiveTypeEntry;)V

第二次尝试:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Could not initialize class org.openx.data.jsonserde.objectinspector.JsonObjectInspectorFactory

• 我可以用这个 SerDe 创建一个表:https://github.com/cloudera/cdh-twitter-example

  我用来自flume的推文创建了一个外部表.我不能做SELECT * FROM tweets;"

  I create an external table with tweets from flume. I can't do "SELECT * FROM tweets;"

  失败:运行时异常 org.apache.hadoop.hive.ql.metadata.HiveException:因异常 java.lang.ClassNotFoundException 失败:com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException:java.lang.ClassNotFoundException:com.cloudera.hive.serde.JSONSerDe

  FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe

  我可以做 SELECT id, text FROM tweets;

  I can do SELECT id, text FROM tweets;

  我无法自行加入此表:

  执行日志位于:/tmp/jochen.debie/jochen.debie_20140311121313_164611a9-b0d8-4e53-9bda-f9f7ac342aaf.log2014-03-11 12:13:30 开始启动本地任务处理地图加入；最大内存 = 257294336执行失败，退出状态:2获取错误信息

  Execution log at: /tmp/jochen.debie/jochen.debie_20140311121313_164611a9-b0d8-4e53-9bda-f9f7ac342aaf.log 2014-03-11 12:13:30 Starting to launch local task to process map join; maximum memory = 257294336 Execution failed with exit status: 2 Obtaining error information

  任务失败！任务编号:第五阶段

  Task failed! Task ID: Stage-5

  提到的执行日志:

  2014-03-11 12:13:30,331 ERROR mr.MapredLocalTask (MapredLocalTask.java:executeFromChildJVM(324)) - Hive Runtime Error: Map local work failed
  org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDejava.lang.RuntimeException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe
  

  有谁知道如何解决这个问题，或者至少告诉我问题出在哪里?

  Does anyone know how to fix this or at least show me where the problem is?

  我在 Hadoop 2.0.0-cdh4.4.0 和 hive 0.10 上构建 serde 会不会有问题?

  Can it be a problem that I built the serde on a Hadoop 2.0.0-cdh4.4.0 and hive 0.10?

  推荐答案

  据我所知，Hive-.11+ 在加入自定义 SerDe 时存在错误.

  From what I've seen, Hive-.11+ has a bug in join with custom SerDe.

  https://github.com/Esri/gis-tools-for-hadoop/issues/9

  您可以尝试将包含 SerDe 类的 JAR 文件复制到 $HIVE_HOME/lib 的解决方法.

  You might try the workaround of copying the JAR file containing the SerDe class, to $HIVE_HOME/lib .

  (我在你的问题中看到你在 join 和其他情况下都遇到了 ClassNotFoundException；到目前为止，我遇到的都是 join.)

  (I see in your question you got ClassNotFoundException both in join and in other cases; so far the times I have encountered such were all with join.)

  另一个解决方法是使用 HADOOP_CLASSPATH:

  Another workaround is to use HADOOP_CLASSPATH:

  env HADOOP_CLASSPATH=some.jar:other.jar hive ...

  [Edit] 变通方法适用于 Hive 版本 0.11 和 0.12；然后 0.13 及更高版本包含对 HIVE-6670 的修复.

  The work around applies to Hive versions 0.11 and 0.12; then 0.13 and above contain the fix for HIVE-6670.

  这篇关于Hive 0.12 和 Hadoop 2.2.0-cdh5.0.0-beta2 的 SerDe 问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！