Spark,在Scala中添加具有相同值的新列 [英] Spark, add new Column with the same value in Scala
问题描述
在Spark-Scala环境中,withColumn
函数存在一些问题.
我想像这样在我的DataFrame中添加一个新列:
I have some problem with the withColumn
function in Spark-Scala environment.
I would like to add a new Column in my DataFrame like that:
+---+----+---+
| A| B| C|
+---+----+---+
| 4|blah| 2|
| 2| | 3|
| 56| foo| 3|
|100|null| 5|
+---+----+---+
成为:
+---+----+---+-----+
| A| B| C| D |
+---+----+---+-----+
| 4|blah| 2| 750|
| 2| | 3| 750|
| 56| foo| 3| 750|
|100|null| 5| 750|
+---+----+---+-----+
D列中的一个值对我的DataFrame中的每一行重复N次.
the column D in one value repeated N-time for each row in my DataFrame.
代码是这样的:
var totVehicles : Double = df_totVehicles(0).getDouble(0); //return 750
变量totVehicles返回正确的值,可以正常工作!
The variable totVehicles returns the correct value, it's works!
第二个DataFrame必须计算2个字段(id_zipcode,n_vehicles),并添加第三列(具有相同的值-750):
The second DataFrame has to calculate 2 fields (id_zipcode, n_vehicles), and add the third column (with the same value -750):
var df_nVehicles =
df_carPark.filter(
substring($"id_time",1,4) < 2013
).groupBy(
$"id_zipcode"
).agg(
sum($"n_vehicles") as 'n_vehicles
).select(
$"id_zipcode" as 'id_zipcode,
'n_vehicles
).orderBy(
'id_zipcode,
'n_vehicles
);
最后,我使用withColumn
函数添加新列:
Finally, I add the new column with withColumn
function:
var df_nVehicles2 = df_nVehicles.withColumn(totVehicles, df_nVehicles("n_vehicles") + df_nVehicles("id_zipcode"))
但是Spark向我返回此错误:
But Spark returns me this error:
error: value withColumn is not a member of Unit
var df_nVehicles2 = df_nVehicles.withColumn(totVehicles, df_nVehicles("n_vehicles") + df_nVehicles("id_zipcode"))
你能帮我吗? 非常感谢你!
Can you help me? Thank you very much!
推荐答案
lit
函数用于将文字值添加为列
lit
function is for adding literal values as a column
import org.apache.spark.sql.functions._
df.withColumn("D", lit(750))
这篇关于Spark,在Scala中添加具有相同值的新列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!