如何在Spark Scala中迭代数据框中的每一列 [英] How to Iterate each column in a Dataframe in Spark Scala

问题描述

假设我有一个包含多列的数据框，我想迭代每一列，进行一些计算并更新该列.有什么好办法吗?

Suppose I have a dataframe with multiple columns, I want to iterate each column, do some calculation and update that column. Is there any good way to do that?

推荐答案

更新 在下面的示例中，我有一个带有两个整数列c1和c2的数据框.每列的值除以其列的总和.

Update In below example I have a dataframe with two integer columns c1 and c2. each column's value is divided with the sum of its columns.

import org.apache.spark.sql.expressions.Window
val df = Seq((1,15), (2,20), (3,30)).toDF("c1","c2")
val result = df.columns.foldLeft(df)((acc, colname) => acc.withColumn(colname, sum(acc(colname)).over(Window.orderBy(lit(1)))/acc(colname)))

输出:

scala> result.show()
+---+------------------+
| c1|                c2|
+---+------------------+
|6.0| 4.333333333333333|
|3.0|              3.25|
|2.0|2.1666666666666665|
+---+------------------+

这篇关于如何在Spark Scala中迭代数据框中的每一列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！