Scala:如何使用Scala替换数据框中的值 [英] Scala: How can I replace value in Dataframes using scala

问题描述

例如，我想将列中等于0.2的所有数字替换为0.如何在Scala中做到这一点?谢谢

For example I want to replace all numbers equal to 0.2 in a column to 0. How can I do that in Scala? Thanks

修改:

|year| make|model| comment            |blank|
|2012|Tesla| S   | No comment         |     | 
|1997| Ford| E350|Go get one now th...|     | 
|2015|Chevy| Volt| null               | null| 

这是我正在尝试将make列中的Tesla更改为S

This is my Dataframe I'm trying to change Tesla in make column to S

推荐答案

注意: 如奥利维尔·吉拉尔多德(Olivier Girardot)所述，此答案并未经过优化，withColumn解决方案是一个可以使用的解决方案(Azeroth2b答案)

Note: As mentionned by Olivier Girardot, this answer is not optimized and the withColumn solution is the one to use (Azeroth2b answer)

该答案已被接受，因此无法删除

Can not delete this answer as it has been accepted

这是我对此的看法:

 val rdd = sc.parallelize(
      List( (2012,"Tesla","S"), (1997,"Ford","E350"), (2015,"Chevy","Volt"))
  )
  val sqlContext = new SQLContext(sc)

  // this is used to implicitly convert an RDD to a DataFrame.
  import sqlContext.implicits._

  val dataframe = rdd.toDF()

  dataframe.foreach(println)

 dataframe.map(row => {
    val row1 = row.getAs[String](1)
    val make = if (row1.toLowerCase == "tesla") "S" else row1
    Row(row(0),make,row(2))
  }).collect().foreach(println)

//[2012,S,S]
//[1997,Ford,E350]
//[2015,Chevy,Volt]

您实际上可以直接在DataFrame上使用map.

You can actually use directly map on the DataFrame.

因此，您基本上检查了列1中的字符串tesla. 如果它是tesla，则将S值用于make，否则将使用第1列的当前值

So you basically check the column 1 for the String tesla. If it's tesla, use the value S for make else you the current value of column 1

然后使用索引(从零开始)(在我的示例中为Row(row(0),make,row(2)))，用行中的所有数据构建一个元组

Then build a tuple with all data from the row using the indexes (zero based) (Row(row(0),make,row(2))) in my example)

可能有更好的方法来做到这一点.我还不太熟悉Spark伞

There is probably a better way to do it. I am not that familiar yet with the Spark umbrella

这篇关于Scala:如何使用Scala替换数据框中的值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！