Spark，在 Scala 中添加具有相同值的新列 [英] Spark, add new Column with the same value in Scala

问题描述

我对 Spark-Scala 环境中的 withColumn 函数有一些问题.我想在我的 DataFrame 中添加一个新列:

I have some problem with the withColumn function in Spark-Scala environment. I would like to add a new Column in my DataFrame like that:

+---+----+---+
|  A|   B|  C|
+---+----+---+
|  4|blah|  2|
|  2|    |  3|
| 56| foo|  3|
|100|null|  5|
+---+----+---+

变成:

+---+----+---+-----+
|  A|   B|  C|  D  |
+---+----+---+-----+
|  4|blah|  2|  750|
|  2|    |  3|  750|
| 56| foo|  3|  750|
|100|null|  5|  750|
+---+----+---+-----+

一个值中的 D 列对于我的 DataFrame 中的每一行重复 N 次.

the column D in one value repeated N-time for each row in my DataFrame.

代码如下:

var totVehicles : Double = df_totVehicles(0).getDouble(0); //return 750

变量 totVehicles 返回正确的值，它有效！

The variable totVehicles returns the correct value, it's works!

第二个DataFrame要计算2个字段(id_zipcode，n_vehicles)，并添加第三列(相同的值-750):

The second DataFrame has to calculate 2 fields (id_zipcode, n_vehicles), and add the third column (with the same value -750):

var df_nVehicles =
df_carPark.filter(
      substring($"id_time",1,4) < 2013
    ).groupBy(
      $"id_zipcode"
    ).agg(
      sum($"n_vehicles") as 'n_vehicles
    ).select(
      $"id_zipcode" as 'id_zipcode,
      'n_vehicles
    ).orderBy(
      'id_zipcode,
      'n_vehicles
    );

最后，我使用 withColumn 函数添加新列:

Finally, I add the new column with withColumn function:

var df_nVehicles2 = df_nVehicles.withColumn(totVehicles, df_nVehicles("n_vehicles") + df_nVehicles("id_zipcode"))

但是 Spark 返回给我这个错误:

But Spark returns me this error:

 error: value withColumn is not a member of Unit
         var df_nVehicles2 = df_nVehicles.withColumn(totVehicles, df_nVehicles("n_vehicles") + df_nVehicles("id_zipcode"))

你能帮我吗?非常感谢！

Can you help me? Thank you very much!

推荐答案

lit 函数用于将字面值添加为列

lit function is for adding literal values as a column

import org.apache.spark.sql.functions._
df.withColumn("D", lit(750))

这篇关于Spark，在 Scala 中添加具有相同值的新列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！