在调用集合操作的 DataFrame 中不能有映射类型列 [英] Cannot have map type columns in DataFrame which calls set operations

问题描述

:org.apache.spark.sql.AnalysisException:DataFrame 中不能有调用集合操作(​​相交、除外等)的映射类型列，但列 map_col 的类型是映射

: org.apache.spark.sql.AnalysisException: Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column map_col is map

我有一个带有类型列的配置单元表 - MAP.当我尝试在 spark 上下文中对该表进行插入时，出现上述错误.在没有distinct"的情况下，插入工作正常.

I have a hive table with a column of type - MAP<Float, Float>. I get the above error when I try to do an insertion on this table in a spark context. Insertion works fine without the 'distinct'.

create table test_insert2(`test_col` string, `map_col` MAP<INT,INT>) 
location 's3://mybucket/test_insert2';

insert into test_insert2 
select distinct 'a' as test_col, map(0,0) as map_col

推荐答案

尝试将 dataframe 转换为 .rdd 然后应用 .distinct函数.

Try to convert dataframe to .rdd then apply .distinct function.

示例:

spark.sql("select 'a'test_col,map(0,0)map_col 
              union all 
          select 'a'test_col,map(0,0)map_col").rdd.distinct.collect

结果:

Array[org.apache.spark.sql.Row] = Array([a,Map(0 -> 0)])

这篇关于在调用集合操作的 DataFrame 中不能有映射类型列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！