在 AZURE hdinsights 中将 12 个嵌套级别的 json 文件读入 hive [英] Read a json file with 12 nested level into hive in AZURE hdinsights

问题描述

我尝试手动为 json 文件创建架构并尝试创建 Hive 表，但我得到了列类型名称长度 10888 超过了最大允许长度 2000.

I tried to create a schema for the json file manually and tried to create a Hive table and i am getting column type name length 10888 exceeds max allowed length 2000.

我猜我必须更改 Metastore 详细信息，但我不确定 azure Hdinsights 中的配置在哪里.

I am guessing i have to change the metastore details but i am not sure where is the config located In azure Hdinsights .

我尝试过的其他方式是我从 spark 数据框中获得了架构，并尝试从视图中创建表，但仍然遇到相同的错误.

Other way I tried was I got the schema from spark dataframe and i tried to create table from the view but still I get the same error.

这是我在 spark 中尝试的步骤

this are the steps i tried in spark

val tne1 = sc.wholeTextFiles("wasb:path").map(x=>x._2)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
val tne2 = sqlContext.read.json(tne1)   
tne2.createOrReplaceTempView("my_temp_table");
sqlContext.sql("create table s  ROW FORMAT SERDE  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES (  'hive.serialization.extend.nesting.levels'='true') as select * from my_temp_table")

我在这一步遇到错误

org.apache.spark.sql.AnalysisException:org.apache.hadoop.hive.ql.metadata.HiveException:InvalidObjectException(消息:无效的列类型名称长度 5448 超过最大允许长度 2000，类型结构

当我尝试持久化或创建 rdd 时，我获得了架构，但处于格式化视图中.即使我得到了完整的视图，我也可能会提取架构.

org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: InvalidObjectException(message:Invalid column type name length 5448 exceeds max allowed length 2000, type struct

when i try to persist or create the rdd i get the schema but in a formatted view . even if i get the full view i might extract the schema .

推荐答案

我通过 Ambari > Hive > Configs > Advanced > Custom hive-site 添加了以下属性:hive.metastore.max.typename.length=14000.现在我可以创建列类型名称最长为 14000 的表

I Added the following property through Ambari > Hive > Configs > Advanced > Custom hive-site: hive.metastore.max.typename.length=14000. and now i am able to create table with column type name upto 14000 length

这篇关于在 AZURE hdinsights 中将 12 个嵌套级别的 json 文件读入 hive的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！