使用Scala在Apache Spark中连接不同RDD的数据集 [英] Concatenating datasets of different RDDs in Apache spark using scala

问题描述

有没有办法将Spark中两个不同的RDD的数据集串联起来?

Is there a way to concatenate datasets of two different RDDs in spark?

要求是-我使用具有相同列名的scala创建两个中间RDD，需要将两个RDD的这些结果组合在一起并缓存该结果以访问UI.如何在此处合并数据集?

Requirement is - I create two intermediate RDDs using scala which has same column names, need to combine these results of both the RDDs and cache the result for accessing to UI. How do I combine the datasets here?

RDD的类型为spark.sql.SchemaRDD

推荐答案

我认为您正在寻找RDD.union

val rddPart1 = ???
val rddPart2 = ???
val rddAll = rddPart1.union(rddPart2)

示例(在Spark-shell上)

Example (on Spark-shell)

val rdd1 = sc.parallelize(Seq((1, "Aug", 30),(1, "Sep", 31),(2, "Aug", 15),(2, "Sep", 10)))
val rdd2 = sc.parallelize(Seq((1, "Oct", 10),(1, "Nov", 12),(2, "Oct", 5),(2, "Nov", 15)))
rdd1.union(rdd2).collect

res0: Array[(Int, String, Int)] = Array((1,Aug,30), (1,Sep,31), (2,Aug,15), (2,Sep,10), (1,Oct,10), (1,Nov,12), (2,Oct,5), (2,Nov,15))

这篇关于使用Scala在Apache Spark中连接不同RDD的数据集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！