如何解决 AnalysisException:Spark 中已解析的属性 [英] How to resolve the AnalysisException: resolved attribute(s) in Spark

问题描述

val rdd = sc.parallelize(Seq(("vskp", Array(2.0, 1.0, 2.1, 5.4)),("hyd",Array(1.5, 0.5, 0.9, 3.7)),("hyd", Array(1.5, 0.5, 0.9, 3.2)),("tvm", Array(8.0, 2.9, 9.1, 2.5))))val df1= rdd.toDF("id", "vals")val rdd1 = sc.parallelize(Seq(("vskp","ap"),("hyd","tel"),("bglr","kkt")))val df2 = rdd1.toDF("id", "state")val df3 = df1.join(df2,df1("id")===df2("id"),"left")

join 操作正常但是当我重用 df2 时，我面临未解决的属性错误

val rdd2 = sc.parallelize(Seq(("vskp", "Y"),("hyd", "N"),("hyd", "N"),("tvm","Y"))))val df4 = rdd2.toDF("id","存在")val df5 = df4.join(df2,df4("id")===df2("id"),"left")

<块引用>

错误:org.apache.spark.sql.AnalysisException:已解析的属性id#426

解决方案

正如我在评论中提到的，它与https://issues.apache.org/jira/browse/SPARK-10925，更具体地说 https://issues.apache.org/jira/browse/SPARK-14948.重用引用会在命名时产生歧义，因此您必须克隆 df - 请参阅 https://issues.apache.org/jira/browse/SPARK-14948 示例.

val rdd = sc.parallelize(Seq(("vskp", Array(2.0, 1.0, 2.1, 5.4)),("hyd",Array(1.5, 0.5, 0.9, 3.7)),("hyd", Array(1.5, 0.5, 0.9, 3.2)),("tvm", Array(8.0, 2.9, 9.1, 2.5))))
val df1= rdd.toDF("id", "vals")
val rdd1 = sc.parallelize(Seq(("vskp","ap"),("hyd","tel"),("bglr","kkt")))
val df2 = rdd1.toDF("id", "state")
val df3 = df1.join(df2,df1("id")===df2("id"),"left")

The join operation works fine but when I reuse the df2 I am facing unresolved attributes error

val rdd2 = sc.parallelize(Seq(("vskp", "Y"),("hyd", "N"),("hyd", "N"),("tvm", "Y")))
val df4 = rdd2.toDF("id","existance")
val df5 = df4.join(df2,df4("id")===df2("id"),"left")

ERROR: org.apache.spark.sql.AnalysisException: resolved attribute(s)id#426

解决方案

As mentioned in my comment, it is related to https://issues.apache.org/jira/browse/SPARK-10925 and, more specifically https://issues.apache.org/jira/browse/SPARK-14948. Reuse of the reference will create ambiguity in naming, so you will have to clone the df - see the last comment in https://issues.apache.org/jira/browse/SPARK-14948 for an example.

这篇关于如何解决 AnalysisException:Spark 中已解析的属性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！