从 IndexedSeq[DataFrame] 转换为 DataFrame? [英] Convert from IndexedSeq[DataFrame] to DataFrame?

问题描述

新手问题，我正在尝试将列添加到现有的 DataFrame 中，我正在使用 Spark 1.4.1

Newbie question , I am try to add columns to exist DataFrame , I am working with Spark 1.4.1

import sqlContext.implicits._
case class Test(rule: Int)

val test = sc.parallelize((1 to 2).map(i => Test(i-i))).toDF
test.registerTempTable("test")
test.show

+----+
|rule|
+----+
|   0|
|   0|
+----+

然后 - 添加列，一列 - OK

Then - add columns, one column - OK

import org.apache.spark.sql.functions.lit
val t1 = test.withColumn("1",lit(0) )
t1.show

+----+-+
|rule|1|
+----+-+
|   0|0|
|   0|0|
+----+-+

当我尝试添加多列时出现问题:

Problem appears when I try to add several columns:

val t1 = (1 to 5).map( i => test.withColumn(i,lit(i) ))
t1.show()

error: value show is not a member of scala.collection.immutable.IndexedSeq[org.apache.spark.sql.DataFrame]

推荐答案

你需要一个 reduce 过程，所以你可以使用 foldLeft 代替 map 以 test 数据框作为初始参数:

You need a reduce process, so instead of using map, you can use foldLeft with test data frame as your initial parameter:

val t1 = (1 to 5).foldLeft(test){ case(df, i) => df.withColumn(i.toString, lit(i))}

t1.show
+----+---+---+---+---+---+
|rule|  1|  2|  3|  4|  5|
+----+---+---+---+---+---+
|   0|  1|  2|  3|  4|  5|
|   0|  1|  2|  3|  4|  5|
+----+---+---+---+---+---+

这篇关于从 IndexedSeq[DataFrame] 转换为 DataFrame?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！