配置单元运行时错误：无法反序列化缩小输入密钥 [英] Hive Runtime Error: Unable to deserialize reduce input key

问题描述

查询

<$ p $我试图运行一个Insert into分区表p> 'set hive.exec.dynamic.partition = true;
set hive.exec.dynamic.partition.mode = nonstrict;
set hive.execution.engine = tez;
INSERT OVERWRITE TABLE table1 PARTITION（date）select col1，CONCAT（COALESCE（substr（Cdate，1,4），' - '），''，COALESCE（substr（Cdate，6,2），' - ' ），''，COALESCE（substr（Cdate，9,2），' - '）），col3，col4，'mobile-data'，data，date
from
（select col1，substr CDATE，1,10）作为Cdate，u.col3作为col3，u.col4作为col4，date，sum（u.col5 + u.col6）作为来自json_table侧视图的数据explode（json_table.mobiledata）oc作为u
where date> = from_unixtime（unix_timestamp（） - 4 * 60 * 60 * 24，'yyyyMMdd'）
group by col1，substr（CDate，1,10），u.col3，u.col4，日期）t;'


错误：

 '引起的：org.apache.hadoop.hive.ql.metadata.HiveException：配置单元运行时错误：无法反序列化减少从x1x50x48x49x53x48x56x49x57x0x1x117x115x101x114x49x0x1x129x51x122x35x1x65x78x84x32x72x65x76x32x83x101x114x118x105x99x101x0x1x99x111x109x46x100x115x105x46x97x110x116x46x115x101x114x118x101x114x0x1x109x111x98x105x108x101x45x100x97x116x97x0x25输入键5与属性{columns = reducinginkkey0，reducinginkkey1，reducinginkkey2，reducinginkkey3，reducinginkkey4，reducinginkkey5，serialization.lib = org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe，serialization.sort.order = ++++++，columns .types = string，string，string，string，string，string}'
  

错误。有谁知道如何照顾这个？

这个查询工作正常，如果我只是使用查询的select部分，不包括插入到分区表中。

解决方案

我可以使用此链接的帮助来解决此问题。 a href =http://documentation.altiscale.com/hive-runtime-error-unable-to-deserialize-input-key =nofollow> http://documentation.altiscale.com/hive-runtime- error-unable-to-deserialize-input-key

我需要的只是一个设置更改。

 'Set hive.optimize.sort.dynamic.partition = false;'
  

一旦我添加此设置，查询（插入分区表）工作正常。

I am trying to run a Insert in to partition table with group by involved

query

'set hive.exec.dynamic.partition=true;
 set hive.exec.dynamic.partition.mode=nonstrict;
 set hive.execution.engine=tez;
 INSERT OVERWRITE TABLE table1 PARTITION (date) select col1,CONCAT(COALESCE(substr(Cdate,1,4),'-'),'',COALESCE(substr(Cdate,6,2),'-'),'',COALESCE(substr(Cdate,9,2),'-')),col3,col4,'mobile-data',data,date 
from
(select col1,substr(CDate,1,10) as Cdate,u.col3 as col3,u.col4 as col4,date,sum(u.col5+u.col6) as data from json_table lateral view explode(json_table.mobiledata) oc as u 
where date >= from_unixtime(unix_timestamp()-4*60*60*24, 'yyyyMMdd')
group by col1,substr(CDate,1,10),u.col3,u.col4,date)t;'

error:

'Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x48x56x49x57x0x1x117x115x101x114x49x0x1x129x51x122x35x1x65x78x84x32x72x65x76x32x83x101x114x118x105x99x101x0x1x99x111x109x46x100x115x105x46x97x110x116x46x115x101x114x118x101x114x0x1x109x111x98x105x108x101x45x100x97x116x97x0x255 with properties{columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=++++++, columns.types=string,string,string,string,string,string}'

I have never seen this error. Does anyone know how to take care of this?

This query is working fine if i just use the select part of the query excluding the insert into partitioned table.

解决方案

I was able to get this resolved using help from this link

http://documentation.altiscale.com/hive-runtime-error-unable-to-deserialize-input-key

All i needed was a setting change.

'Set hive.optimize.sort.dynamic.partition = false;'

Once i add this setting, query(insert into partitioned table) is working fine.

这篇关于配置单元运行时错误：无法反序列化缩小输入密钥的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！