删除重复项而不会打乱Spark [英] Remove Duplicates without shuffle Spark

问题描述

我有一个带有列的Cassandra表XYX(uuid，插入时间戳，标题文字)

I have a Cassandra table XYX with columns( id uuid, insert a timestamp, header text)

id和insert是复合主键.

Where id and insert are composite primary key.

我正在使用Dataframe，并且在我的Spark Shell中，我正在获取id和header列.我想根据ID和标题列创建不同的行.

I'm using Dataframe and in my spark shell I'm fetching id and header column. I want to have distinct rows based on id and header column.

我看到了很多洗牌，因为Spark Cassandra连接器确保给定Cassandra分区的所有行都在同一个Spark分区中，所以情况并非如此.

I'm seeing lot of shuffles which not be the case since Spark Cassandra connector ensures that all rows for a given Cassandra partition are in same spark partition.

获取后，我使用dropDuplicates来获取不同的记录.

After fetching I'm using dropDuplicates to get distinct records.

推荐答案

Spark Dataframe API尚不支持自定义分区程序.因此，连接器无法将C *分区程序引入Dataframe引擎.RDD Spark API从另一方面支持自定义分区.因此，您可以将数据加载到RDD中，然后将其隐藏到df中.这是有关C *分区程序用法的连接器文档: https://github.com/datastax/spark-cassandra-connector/blob/master/doc/16_partitioning.md

Spark Dataframe API does not support custom partitioners yet. So the Connector could not introduce the C* partitioner to Dataframe engine. An RDD Spark API supports custom partitioner from other hand. Thus you could load your data into RDD and then covert it to df. Here is a Connector doc about C* partitioner usage: https://github.com/datastax/spark-cassandra-connector/blob/master/doc/16_partitioning.md

keyBy()函数可让您定义用于分组的键列

keyBy() function allow you to define key columns to use for grouping

这是工作示例.时间不短，所以我希望有人可以改善它:

Here is working example. It is not short, so I expect someone could improve it:

//load data into RDD and define a group key
val rdd = sc.cassandraTable[(String, String)] ("test", "test")
   .select("id" as "_1", "header" as "_2")
   .keyBy[Tuple1[Int]]("id")
// check that partitioner is CassandraPartitioner
rdd.partitioner
// call distinct for each group, flat it, get two column DF
val df = rdd.groupByKey.flatMap {case (key,group) => group.toSeq.distinct}
    .toDF("id", "header")

这篇关于删除重复项而不会打乱Spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！