删除重复行，无论新信息如何 -PySpark [英] Remove duplicate rows, regardless of new information -PySpark

问题描述

假设我有一个像这样的数据框:

Say I have a dataframe like so:

ID         Media
1         imgix.com/20830dk
2         imgix.com/202398pwe
3         imgix.com/lvw0923dk
4         imgix.com/082kldcm
4         imgix.com/lks032m
4         imgix.com/903248

我想结束:

ID         Media
1         imgix.com/20830dk
2         imgix.com/202398pwe
3         imgix.com/lvw0923dk
4         imgix.com/082kldcm

即使这导致我丢失了 ID = 4 的 2 个链接，但我不在乎.在 python/pyspark 中有没有一种简单的方法可以做到这一点?

Even though that causes me to lose 2 links for ID = 4, I don't care. Is there a simple way to do this in python/pyspark?

推荐答案

Anton 和 pault 是正确的:

Anton and pault are correct:

df.drop_duplicates(subset=['ID']) 

确实有效

这篇关于删除重复行，无论新信息如何 -PySpark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！