Spark 1.5.2:org.apache.spark.sql.AnalysisException:未解决的运算符'Union; [英] Spark 1.5.2: org.apache.spark.sql.AnalysisException: unresolved operator 'Union;

问题描述

我有两个数据框 df1 和 df2 .他们两个都具有以下架构:

I have two dataframes df1 and df2. Both of them have the following schema:

 |-- ts: long (nullable = true)
 |-- id: integer (nullable = true)
 |-- managers: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- projects: array (nullable = true)
 |    |-- element: string (containsNull = true)

df1 是从avro文件中创建的，而 df2 是从等效的镶木地板文件中创建的.但是，如果执行 df1.unionAll(df2).show()，则会出现以下错误:

df1 is created from an avro file while df2 from an equivalent parquet file. However, If I execute, df1.unionAll(df2).show(), I get the following error:

    org.apache.spark.sql.AnalysisException: unresolved operator 'Union;
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:174)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49)
    at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103)

推荐答案

我遇到了相同的情况，结果不仅是字段必须相同，而且还需要保持字段的顺序完全相同.两个数据框中的字段，以使其正常工作.

I ran into the same situation and it turns out to be not only the fields need to be the same but also you need to maintain the exact same ordering of the fields in both dataframe in order to make it work.

这篇关于Spark 1.5.2:org.apache.spark.sql.AnalysisException:未解决的运算符'Union;的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！