从IndexedSeq [DataFrame]转换为DataFrame? [英] Convert from IndexedSeq[DataFrame] to DataFrame?

问题描述

新手问题， 我正在尝试添加列以存在DataFrame，我正在使用Spark 1.4.1

Newbie question , I am try to add columns to exist DataFrame , I am working with Spark 1.4.1

import sqlContext.implicits._
case class Test(rule: Int)

val test = sc.parallelize((1 to 2).map(i => Test(i-i))).toDF
test.registerTempTable("test")
test.show

+----+
|rule|
+----+
|   0|
|   0|
+----+

然后-添加列，一列-确定

Then - add columns, one column - OK

import org.apache.spark.sql.functions.lit
val t1 = test.withColumn("1",lit(0) )
t1.show

+----+-+
|rule|1|
+----+-+
|   0|0|
|   0|0|
+----+-+

当我尝试添加几列时出现问题:

Problem appears when I try to add several columns:

val t1 = (1 to 5).map( i => test.withColumn(i,lit(i) ))
t1.show()

error: value show is not a member of scala.collection.immutable.IndexedSeq[org.apache.spark.sql.DataFrame]

推荐答案

您需要 reduce 流程，因此可以使用 foldLeft代替使用 map ，其中 test 数据框为初始参数:

You need a reduce process, so instead of using map, you can use foldLeft with test data frame as your initial parameter:

val t1 = (1 to 5).foldLeft(test){ case(df, i) => df.withColumn(i.toString, lit(i))}

t1.show
+----+---+---+---+---+---+
|rule|  1|  2|  3|  4|  5|
+----+---+---+---+---+---+
|   0|  1|  2|  3|  4|  5|
|   0|  1|  2|  3|  4|  5|
+----+---+---+---+---+---+

这篇关于从IndexedSeq [DataFrame]转换为DataFrame?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！