spark:如何在保持最高时间戳行的同时对数据帧执行 dropDuplicates [英] spark: How to do a dropDuplicates on a dataframe while keeping the highest timestamped row

问题描述

我有一个用例，我需要删除数据帧的重复行(在这种情况下，重复意味着它们具有相同的id"字段)，同时保留具有最高时间戳"(unix 时间戳)字段的行.

I have a use case where I'd need to drop duplicate rows of a dataframe (in this case duplicate means they have the same 'id' field) while keeping the row with the highest 'timestamp' (unix timestamp) field.

我找到了 drop_duplicate 方法(我使用的是 pyspark)，但无法控制将保留哪个项目.

I found the drop_duplicate method (I'm using pyspark), but one don't have control on which item will be kept.

有人可以帮忙吗?提前谢谢

Anyone can help ? Thx in advance

推荐答案

可能需要手动 map 和 reduce 来提供您想要的功能.

A manual map and reduce might be needed to provide the functionality you want.

def selectRowByTimeStamp(x,y):
    if x.timestamp > y.timestamp:
        return x
    return y

dataMap = data.map(lambda x: (x.id, x))
uniqueData = dataMap.reduceByKey(selectRowByTimeStamp) 

这里我们根据 id 对所有数据进行分组.然后，当我们减少分组时，我们会保留具有最高时间戳的记录.代码完成reduce后，每个id只剩下1条记录.

Here we are grouping all of the data based on id. Then, when we are reducing the groupings, we do so by keeping the record with the highest timestamp. When the code is done reducing, only 1 record will be left for each id.

这篇关于spark:如何在保持最高时间戳行的同时对数据帧执行 dropDuplicates的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！