Spark Scala中的歧义架构 [英] Ambiguous schema in Spark Scala

问题描述

模式:

|-- c0: string (nullable = true)
|-- c1: struct (nullable = true)
|    |-- c2: array (nullable = true)
|    |    |-- element: struct (containsNull = true)
|    |    |    |-- orangeID: string (nullable = true)
|    |    |    |-- orangeId: string (nullable = true)

我试图在火花中展平上面的模式.

I am trying to flatten the schema above in spark.

代码:

var df = data.select($"c0",$"c1.*").select($"c0",explode($"c2")).select($"c0",$"col.orangeID", $"col.orangeId")

展平代码工作正常.问题出在最后一部分，其中两列之间仅相差1个字母(orangeID和orangeId).因此，我收到此错误:

The flattening code is working fine. The problem is in the last part where the 2 columns differ only by 1 letter (orangeID and orangeId). Hence I am getting this error:

错误:

org.apache.spark.sql.AnalysisException: Ambiguous reference to fields StructField(orangeID,StringType,true), StructField(orangeId,StringType,true);

任何避免这种歧义的建议都是很好的.

Any suggestions to avoid this ambiguity will be great.

推荐答案

打开spark sql区分大小写的配置并尝试

turn on the spark sql case sensitivity configuration and try

spark.sql("set spark.sql.caseSensitive=true")

这篇关于Spark Scala中的歧义架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！