Scala通过表达式将新列添加到数据框 [英] Scala add new column to dataframe by expression

问题描述

我将使用表达式将新列添加到数据框. 例如，我的数据框为

I am going to add new column to a dataframe with expression. for example, I have a dataframe of

+-----+----------+----------+-----+
| C1  | C2       |   C3     |C4   |
+-----+----------+----------+-----+
|steak|1         |1         |  150|
|steak|2         |2         |  180|
| fish|3         |3         |  100|
+-----+----------+----------+-----+

并且我想创建一个新的表达式为"C2/C3 + C4"的列C5，假设需要添加几个新列，并且这些表达式可能不同并且来自数据库.

and I want to create a new column C5 with expression "C2/C3+C4", assuming there are several new columns need to add, and the expressions may be different and come from database.

有什么好方法吗?

我知道，如果我有一个类似"2 + 3 * 4"的表达式，则可以使用scala.tools.reflect.ToolBox进行评估.

I know that if I have an expression like "2+3*4" I can use scala.tools.reflect.ToolBox to eval it.

通常我使用df.withColumn添加新列.

And normally I am using df.withColumn to add new column.

似乎我需要创建一个UDF，但是如何将列值作为参数传递给UDF?特别是可能有多个表达式需要不同的列来计算.

Seems I need to create an UDF, but how can I pass the columns value as parameters to UDF? especially there maybe multiple expression need different columns calculate.

推荐答案

可以使用expr从表达式创建Column来完成:

This can be done using expr to create a Column from an expression:

val df = Seq((1,2)).toDF("x","y")

val myExpression = "x+y"

import org.apache.spark.sql.functions.expr

df.withColumn("z",expr(myExpression)).show()

+---+---+---+
|  x|  y|  z|
+---+---+---+
|  1|  2|  3|
+---+---+---+

这篇关于Scala通过表达式将新列添加到数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！