使用/不使用 Spark SQL 连接两个普通 RDD [英] Join two ordinary RDDs with/without Spark SQL

问题描述

我需要在一列/多列上加入两个普通的RDD.逻辑上这个操作相当于两个表的数据库join操作.我想知道这是否只能通过 Spark SQL 或其他方法来实现.

I need to join two ordinary RDDs on one/more columns. Logically this operation is equivalent to the database join operation of two tables. I wonder if this is possible only through Spark SQL or there are other ways of doing it.

作为一个具体的例子，考虑RDD r1 主键 ITEM_ID:

As a concrete example, consider RDD r1 with primary key ITEM_ID:

(ITEM_ID, ITEM_NAME, ITEM_UNIT, COMPANY_ID)

和带有主键COMPANY_ID的RDD r2:

and RDD r2 with primary key COMPANY_ID:

(COMPANY_ID, COMPANY_NAME, COMPANY_CITY)

我想加入r1 和r2.

如何做到这一点?

推荐答案

Soumya Simanta 给出了很好的答案.但是joined RDD中的值是Iterable，所以结果可能和普通table join不太相似.

Soumya Simanta gave a good answer. However, the values in joined RDD are Iterable, so the results may not be very similar to ordinary table joining.

或者，您可以:

val mappedItems = items.map(item => (item.companyId, item))
val mappedComp = companies.map(comp => (comp.companyId, comp))
mappedItems.join(mappedComp).take(10).foreach(println)

输出将是:

(c1,(Item(1,first,2,c1),Company(c1,company-1,city-1)))
(c1,(Item(2,second,2,c1),Company(c1,company-1,city-1)))
(c2,(Item(3,third,2,c2),Company(c2,company-2,city-2)))

这篇关于使用/不使用 Spark SQL 连接两个普通 RDD的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！