将行转换为火花 Scala 中的地图 [英] Convert Row to map in spark scala

问题描述

我有一个数据框中的一行，我想将其转换为 Map[String, Any]，该 Map[String, Any] 将列名称映射到该列的行中的值.

I have a row from a data frame and I want to convert it to a Map[String, Any] that maps column names to the values in the row for that column.

有简单的方法吗?

我是为像

def rowToMap(row:Row): Map[String, String] = {
row.schema.fieldNames.map(field => field -> row.getAs[String](field)).toMap
}

val myRowMap = rowToMap(myRow)

如果该行包含其他值，而不是像 String 这样的特定值，那么代码会变得更加混乱，因为该行没有方法 .get(field)

If the row contains other values, not specific ones like String then the code gets messier because the row does not have a a method .get(field)

有什么想法吗?

推荐答案

您可以使用 getValuesMap:

val df = Seq((1, 2.0, "a")).toDF("A", "B", "C")    
val row = df.first

获取Map[String, Any]:

row.getValuesMap[Any](row.schema.fieldNames)
// res19: Map[String,Any] = Map(A -> 1, B -> 2.0, C -> a)

或者你可以得到 Map[String, AnyVal] 这个简单的例子，因为值不是复杂的对象

Or you can get Map[String, AnyVal] for this simple case since the values are not complex objects

row.getValuesMap[AnyVal](row.schema.fieldNames)
// res20: Map[String,AnyVal] = Map(A -> 1, B -> 2.0, C -> a)

注意:_{getValuesMap的返回值类型可以标注为任意类型，所以你不能依靠它来判断你的数据类型有但需要记住你从一开始就拥有什么.}

Note: _{the returned value type of the getValuesMap can be labelled as any type, so you can not rely on it to figure out what data types you have but need to keep in mind what you have from the beginning instead.}

这篇关于将行转换为火花 Scala 中的地图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！