替换火花数据帧中所有列名中的空格 [英] Replacing whitespace in all column names in spark Dataframe

问题描述

我有一些列名中带有空格的 spark 数据框，必须用下划线替换.

I have spark dataframe with whitespaces in some of column names, which has to be replaced with underscore.

我知道可以在 sparkSQL 中使用 withColumnRenamed() 重命名单个列，但是要重命名 'n' 个列，此函数必须链接 'n' 次(据我所知).

I know a single column can be renamed using withColumnRenamed() in sparkSQL, but to rename 'n' number of columns, this function has to chained 'n' times (to my knowledge).

为了自动执行此操作，我尝试过:

To automate this, i have tried:

val old_names = df.columns()        // contains array of old column names

val new_names = old_names.map { x => 
   if(x.contains(" ") == true) 
      x.replaceAll("\\s","_") 
   else x 
}                    // array of new column names with removed whitespace.

现在，如何用 new_names

推荐答案

  var newDf = df
  for(col <- df.columns){
    newDf = newDf.withColumnRenamed(col,col.replaceAll("\\s", "_"))
  }

你可以用某种方法封装它，这样就不会造成太大的污染.

You can encapsulate it in some method so it won't be too much pollution.

这篇关于替换火花数据帧中所有列名中的空格的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！