如何在缺少名称的情况下向DataFrame添加新列? [英] How to add new columns to DataFrame given their names when they are missing?

问题描述

我想向DataFrame添加尚不可用的选定列.

I'd like to add selected columns to a DataFrame that are not available already.

val columns=List("Col1","Col2","Col3") 
for(i<-columns) 
 if(!df.schema.fieldNames.contains(i)==true)
 df.withColumn(i,lit(0))

选择列时，数据框仅旧列会出现，新列将不会出现.

When select column the data frame only old column are coming, new columns are not coming.

推荐答案

与Spark相比，它更多地是在Scala中进行操作，对于foldLeft(我最喜欢的！)是一个很好的例子.

It's more about how to do it in Scala than Spark and is excellent case for foldLeft (my favorite!)

// start with an empty DataFrame, but could be anything
val df = spark.emptyDataFrame
val columns = Seq("Col1", "Col2", "Col3")
val columnsAdded = columns.foldLeft(df) { case (d, c) =>
  if (d.columns.contains(c)) {
    // column exists; skip it
    d
  } else {
    // column is not available so add it
    d.withColumn(c, lit(0))
  }
}

scala> columnsAdded.printSchema
root
 |-- Col1: integer (nullable = false)
 |-- Col2: integer (nullable = false)
 |-- Col3: integer (nullable = false)

这篇关于如何在缺少名称的情况下向DataFrame添加新列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！