配置单元查询执行:失败，发生异常java.io.IOException:org.apache.avro.AvroTypeException:发现了两次，期望合并 [英] Hive query execution: Failed with exception java.io.IOException:org.apache.avro.AvroTypeException: Found double, expecting union

问题描述

我正在尝试执行一个简单的

select * from table limit 1;

配置单元中外部表上的语句.但是面对执行失败:

java.io.IOException:org.apache.avro.AvroTypeException: Found double, expecting union. 

有人可以帮助我理解这是什么意思吗?我已经检查了模式文件，并且已经给出了"default":null.

此异常发生的确切原因是什么?

我试图理解已经存在的讨论.

架构看起来像这样:

{"type":"record",
 "name":"VISIBILITY",
 "namespace":"pentaho_etl",
 "fields":[ {"name":"ID",               "type":["null","long"],     "default":null},
            {"name":"VERSION_ID",       "type":["null","long"],     "default":null},
            {"name":"IMP_CLASS",        "type":["null","string"],   "default":null},
            {"name":"NAME",             "type":["null","string"],   "default":null},
            {"name":"DL_SCHEMA_VERSN",  "type":["null","long"],     "default":null},
            {"name":"DL_FEED_TM",       "type":["null","string"],   "default":null},
            {"name":"DL_WFID",          "type":["null","long"],     "default":null},
            {"name":"DL_ENT_NM",        "type":["null","string"],   "default":null},
            {"name":"DL_JID",           "type":["null","long"],     "default":null},
            {"name":"DL_DATASET_ID",    "type":["null","long"],     "default":null},
            {"name":"DL_FD_DT",         "type":["null","string"],   "default":null}
        ]
}

解决方案

我知道，这是一个老问题，但尚未得到答案，因此，我几乎不了解有关此问题的信息，因此对其他人有帮助./p>

这是一个非常普遍的问题，由于表架构与avro文件中包含的架构之间的不匹配而导致我们面临.

尽管avro文件是二进制文件，但是您可以在任何文本编辑器中将其打开，并从最初的几行中获取其架构.此架构与表的架构不匹配，可以使用show create table table_name来获得该架构. 然后，您可以逐列比较这两种架构，以查找任何差异，例如您的avro将一列显示为双列，但您的架构可能会将其显示为字符串.

I am trying to execute a simple

select * from table limit 1;

Statement in hive on an external table. But facing failue with execption:

java.io.IOException:org.apache.avro.AvroTypeException: Found double, expecting union. 

Can Someone help me understand what this means? I have checked the schema file and the "default":null is already given.

What is the exact reason for this exception occuring?

I tried understanding an already existing discussion.

The schema looks something like this:

{"type":"record",
 "name":"VISIBILITY",
 "namespace":"pentaho_etl",
 "fields":[ {"name":"ID",               "type":["null","long"],     "default":null},
            {"name":"VERSION_ID",       "type":["null","long"],     "default":null},
            {"name":"IMP_CLASS",        "type":["null","string"],   "default":null},
            {"name":"NAME",             "type":["null","string"],   "default":null},
            {"name":"DL_SCHEMA_VERSN",  "type":["null","long"],     "default":null},
            {"name":"DL_FEED_TM",       "type":["null","string"],   "default":null},
            {"name":"DL_WFID",          "type":["null","long"],     "default":null},
            {"name":"DL_ENT_NM",        "type":["null","string"],   "default":null},
            {"name":"DL_JID",           "type":["null","long"],     "default":null},
            {"name":"DL_DATASET_ID",    "type":["null","long"],     "default":null},
            {"name":"DL_FD_DT",         "type":["null","string"],   "default":null}
        ]
}

解决方案

I know, it is old question, but it is unanswered, hence, adding little information that I know about this issue, so that, it helps others.

This is very common issue, we face due to mismatch between the table schema and schema contained in the avro file.

Though, avro file is binary, you can open it in any text editor and get its schema from initial few lines. This schema is not matching with the schema of the table, which you can get using show create table table_name. Then you can compare these 2 schemas column by column to find any discrepancies e.g. your avro is showing a column as double but your schema might have it as string.

这篇关于配置单元查询执行:失败，发生异常java.io.IOException:org.apache.avro.AvroTypeException:发现了两次，期望合并的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！