如何根据其列的子集有效地选择 RDD 上的不同行 [英] How to efficiently select distinct rows on an RDD based on a subset of its columns`

问题描述

考虑一个案例类:

case class Prod(productId: String, date: String, qty: Int, many other attributes ..)

还有一个

val rdd: RDD[Prod]

包含该类的许多实例.

唯一键是 (productId,date) 元组.但是我们确实有一些重复.

The unique key is intended to be the (productId,date) tuple. However we do have some duplicates.

有什么有效的方法可以去除重复项吗?

Is there any efficient means to remove the duplicates?

操作

      rdd.distinct

将查找重复的整行.

后备将涉及将唯一的 (productId,date) 组合连接回整行:我正在研究如何做到这一点.但即便如此，它也有几个操作.如果存在更简单的方法(也更快?)会很有用.

A fallback would involve joining the unique (productId,date) combinations back to the entire rows: I am working through exactly how to do this. But even so it is several operations. A simpler approach (faster as well?) would be useful if it exists.

推荐答案

我会在 Dataset 上使用 dropDuplicates:

val rdd = sc.parallelize(Seq(
  Prod("foo", "2010-01-02", 1), Prod("foo", "2010-01-02", 2)
))

rdd.toDS.dropDuplicates("productId", "date")

但是 reduceByKey 应该也能工作:

but reduceByKey should work as well:

rdd.keyBy(prod => (prod.productId, prod.date)).reduceByKey((x, _) => x).values

这篇关于如何根据其列的子集有效地选择 RDD 上的不同行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！