失败,出现异常java.io.IOException:org.apache.avro.AvroTypeException:发现很久,期望在配置单元中工会 [英] Failed with exception java.io.IOException:org.apache.avro.AvroTypeException: Found long, expecting union in hive
问题描述
需要帮助!!!
我使用 flume
将twitter feed送入hdfs并加载 hive
进行分析。
步骤如下:
hdfs 中的数据:
我已经在 avsc
文件中描述了 avro schema
并将其放入hadoop:
{type:record,
name:Doc,
doc:adoc ,
fields:[{name:id,type:string},
{name:user_friends_count,type:[int ,null]},
{name:user_location,type:[string,null]},
{name:user_description,type :[string,null]},
{name:user_statuses_count,type:[int,null]},
{name user_followers_count,type:[int,null]},
{name:user_name,type:[string,null]},
{name:user_screen_name,type:[string,null]},
{name:created_at,type:[string,null]},
{name:text,type:[string,null] },
{name:retweet_count,type:[boolean,null]},
{name:retweeted,type:[ ,null]},
{name:in_reply_to_user_id,type:[long,null]},
{name:source类型:[string,null]},
{name:in_reply_to_status_id,type:[long,null]},
{ :media_url_https,type:[string,null]},
{name:expanded_url,type:[string,null]}]}
我写了一个.hql文件来创建一个表并在其中加载数据:
create table tweetsavro
行格式serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
以inputformat存储
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
outputformat
'org.apache.hadoop.hive.ql.io.avro。 AvroContainerOutputFormat '
tblproperties('avro.schema.url'='hdfs:///avro_schema/AvroSchemaFile.avsc');
载入数据inpath'/test/twitter_data/FlumeData.*'覆盖到表格tweetsavro;
我已成功运行.hql文件,但是当我运行 select *从配置单元中的< tablename>
命令显示以下错误:
tweetsavro的输出是:
hive> desc tweetsavro;
OK
id字符串
user_friends_count int
user_location字符串
user_description字符串
user_statuses_count int
user_followers_count int
user_name字符串
user_screen_name字符串
created_at字符串
文本字符串
retweet_count布尔
转发布尔
in_reply_to_user_id bigint
源s tring
in_reply_to_status_id bigint
media_url_https字符串
expanded_url字符串
所用时间:0.697秒,提取:17行
我面对完全相同的问题。该问题存在于时间戳字段(您的案例中的created_at列),我试图将字符串插入到我的新表中。我的假设是这个数据将在我的源代码中的 [null,string]
格式。我分析了从sqoop import --as-avrodatafile进程生成的源avro模式。根据导入生成的avro模式在timestamp列中具有以下签名。
{
name:order_date,
type:[ null,long],
default:null,
columnName:order_date,
sqlType:93
} code>
SqlType 93代表Timestamp数据类型。因此,在我的目标表Avro模式文件中,我将数据类型更改为'long',这解决了问题。我的猜测可能是你列中的数据类型不匹配。
Need help!!!
I am streaming twitter feeds into hdfs using flume
and loading it up in hive
for analysis.
The steps are as follows:
Data in hdfs:
I have described the avro schema
in an avsc
file and put it in hadoop:
{"type":"record",
"name":"Doc",
"doc":"adoc",
"fields":[{"name":"id","type":"string"},
{"name":"user_friends_count","type":["int","null"]},
{"name":"user_location","type":["string","null"]},
{"name":"user_description","type":["string","null"]},
{"name":"user_statuses_count","type":["int","null"]},
{"name":"user_followers_count","type":["int","null"]},
{"name":"user_name","type":["string","null"]},
{"name":"user_screen_name","type":["string","null"]},
{"name":"created_at","type":["string","null"]},
{"name":"text","type":["string","null"]},
{"name":"retweet_count","type":["boolean","null"]},
{"name":"retweeted","type":["boolean","null"]},
{"name":"in_reply_to_user_id","type":["long","null"]},
{"name":"source","type":["string","null"]},
{"name":"in_reply_to_status_id","type":["long","null"]},
{"name":"media_url_https","type":["string","null"]},
{"name":"expanded_url","type":["string","null"]}]}
I have written an .hql file to create a table and loaded data in it:
create table tweetsavro
row format serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
stored as inputformat
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
outputformat
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
tblproperties ('avro.schema.url'='hdfs:///avro_schema/AvroSchemaFile.avsc');
load data inpath '/test/twitter_data/FlumeData.*' overwrite into table tweetsavro;
I have successfully run the .hql file but when i run the select *from <tablename>
command in hive it shows the following error:
The output of tweetsavro is:
hive> desc tweetsavro;
OK
id string
user_friends_count int
user_location string
user_description string
user_statuses_count int
user_followers_count int
user_name string
user_screen_name string
created_at string
text string
retweet_count boolean
retweeted boolean
in_reply_to_user_id bigint
source string
in_reply_to_status_id bigint
media_url_https string
expanded_url string
Time taken: 0.697 seconds, Fetched: 17 row(s)
I was facing the exact same issue. The issue existed in the timestamp field("created_at" column in your case) which i was trying to insert as string into my new table. My assumption was this data would be in [ "null","string"]
format in my source. I analyzed the source avro schema which got generated from the sqoop import --as-avrodatafile process. The avro schema generated from import had the below signature for the timestamp column.
{
"name" : "order_date",
"type" : [ "null", "long" ],
"default" : null,
"columnName" : "order_date",
"sqlType" : "93"
},
SqlType 93 stands for Timestamp datatype. So in my target table Avro Schema file I changed the data type to 'long' and this solved the issue. My guess is possibly the mismatch of datatype in one of your columns.
这篇关于失败,出现异常java.io.IOException:org.apache.avro.AvroTypeException:发现很久,期望在配置单元中工会的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!