使用 Map 替换 Spark 中的列值 [英] Use Map to replace column values in Spark

问题描述

我必须将列列表映射到 Spark 数据集中的另一列:想想这样的事情

I have to map a list of columns to another column in a Spark dataset: think something like this

val translationMap: Map[Column, Column] = Map(
  lit("foo") -> lit("bar"),
  lit("baz") -> lit("bab")
)

我有一个这样的数据框:

And I have a dataframe like this one:

val df = Seq("foo", "baz").toDF("mov")

所以我打算这样翻译:

df.select(
  col("mov"),
  translationMap(col("mov"))
)

但是这段代码吐出以下错误

but this piece of code spits the following error

key not found: movs
java.util.NoSuchElementException: key not found: movs

有没有办法在不连接数百个 when 的情况下执行这种转换?认为 translationMap 可以有很多对键值.

Is there a way to perform such translation without concatenating hundreds of whens? think that translationMap could have lots of pairs key-value.

推荐答案

你应该使用包含地图文字的 Column 而不是 Map[Column, Column]:

Instead of Map[Column, Column] you should use a Column containing a map literal:

import org.apache.spark.sql.functions.typedLit

val translationMap: Column = typedLit(Map(
  "foo" -> "bar",
  "baz" -> "bab"
))

您的其余代码可以保持原样:

The rest of your code can stay as-is:

df.select(
  col("mov"),
  translationMap(col("mov"))
).show

+---+---------------------------------------+
|mov|keys: [foo,baz], values: [bar,bab][mov]|
+---+---------------------------------------+
|foo|                                    bar|
|baz|                                    bab|
+---+---------------------------------------+

这篇关于使用 Map 替换 Spark 中的列值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！