如何在Spark-SQL查询中引用映射列? [英] How to refer a map column in a spark-sql query?

问题描述

scala> val map1 = spark.sql("select map('p1', 's1', 'p2', 's2')")

map1:org.apache.spark.sql.DataFrame = [map(p1，s1，p2，s2):map< string，string>]

map1: org.apache.spark.sql.DataFrame = [map(p1, s1, p2, s2): map<string,string>]

scala> map1.show()

+--------------------+
| map(p1, s1, p2, s2)|
+--------------------+
|[p1 -> s1, p2 -> s2]|
+--------------------+
scala> spark.sql("select element_at(map1, 'p1')")

org.apache.spark.sql.AnalysisException:无法解析给定的" map1 "输入列:[];第1行pos 18;'项目[unresolvedalias('element_at('map1，p1)，无)]

org.apache.spark.sql.AnalysisException: cannot resolve 'map1' given input columns: []; line 1 pos 18; 'Project [unresolvedalias('element_at('map1, p1), None)]

我们如何在第二个SQL查询中重用数据框map1?

How can we reuse the dataframe map1 in second sql query?

推荐答案

map1 是一个具有一列map类型的数据框.此列的名称为 map(p1，s1，p2，s2).例如，可以使用

map1 is a dataframe with a single column of type map. This column has the name map(p1, s1, p2, s2). The dataframe can be queried for example with selectExpr:

map1.selectExpr("element_at(`map(p1, s1, p2, s2)`, 'p1')").show()

打印

+-----------------------------------+
|element_at(map(p1, s1, p2, s2), p1)|
+-----------------------------------+
|                                 s1|
+-----------------------------------+

另一个选择是将数据框注册为临时视图，然后使用sql查询:

Another option is to register the dataframe as temporary view and then use a sql query:

map1.createOrReplaceTempView("map1")
spark.sql("select element_at(`map(p1, s1, p2, s2)`, 'p1') from map1").show()

将显示相同的结果.

这篇关于如何在Spark-SQL查询中引用映射列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！