通过遍历 Scala 列名列表中的列，从 Spark 数据框中删除多个列 [英] Dropping multiple columns from Spark dataframe by Iterating through the columns from a Scala List of Column names

问题描述

我有一个包含大约 400 列的数据框，我想根据我的要求删除 100 列.所以我创建了一个包含 100 个列名的 Scala 列表.然后我想遍历 for 循环以实际删除每个 for 循环迭代中的列.

I have a dataframe which has columns around 400, I want to drop 100 columns as per my requirement. So i have created a Scala List of 100 column names. And then i want to iterate through a for loop to actually drop the column in each for loop iteration.

下面是代码.

final val dropList: List[String] = List("Col1","Col2",...."Col100")

def drpColsfunc(inputDF: DataFrame): DataFrame = { 
    for (i <- 0 to dropList.length - 1) {
        val returnDF = inputDF.drop(dropList(i))
    }
    return returnDF
}

val test_df = drpColsfunc(input_dataframe) 

test_df.show(5)

推荐答案

答案:

val colsToRemove = Seq("colA", "colB", "colC", etc) 

val filteredDF = df.select(df.columns .filter(colName => !colsToRemove.contains(colName)) .map(colName => new Column(colName)): _*) 

这篇关于通过遍历 Scala 列名列表中的列，从 Spark 数据框中删除多个列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！