从对象列表中删除重复项 [英] Remove duplicates from list of object

问题描述

我有MyObject，其字段为:id，a，b，c，e，f 并且我有500 000个项目的列表，现在如何删除参数a，c，f具有相同值的所有重复项目?

I have MyObject with field: id, a, b, c, e, f and I have List with 500 000 items, now how can I remove all duplicate items with of the same value of the parameter a, c, f?

我只在寻找最快，最有效的方法.

I am looking for only the fastest and most efficient method.

更新
我实现了比较器

UPDATE
I implemented comparator

我的班级中的字段具有不同的类型，因此我使用ToString().这是好方法吗?
IdLoc，IdMet，Ser长吗?
Value是对象
IdDataType长

Fields in my class are of different types so I use ToString(). It is good way?
IdLoc, IdMet, Ser are long?
Value is Object
IdDataType is long

class Comparer : IEqualityComparer<MyObject>
{
    public bool Equals(MyObject x, MyObject y)
    {                
        return x.IdLoc == y.IdLoc && x.IdMet == y.IdMet && x.Ser == y.Ser &&
               x.IdDataType == y.IdDataType && x.Time == y.Time && x.Value == y.Value;
    }

    public int GetHashCode(MyObject obj)
    {
        string idLoc = obj.IdLoc.HasValue ? obj.IdLoc.ToString() : String.Empty;
        string idMet = obj.IdMet.HasValue ? obj.IdMet.ToString() : String.Empty;
        string ser = obj.Ser.HasValue ? obj.Ser.ToString() : String.Empty;
        string value = obj.Value != null ?  obj.Value.ToString() : String.Empty;

        return (idLoc + idMet + ser + value + obj.IdDataType.ToString() + obj.Time.ToString()).GetHashCode();
    }
}

删除重复项
元素566890
1)时间:2秒

Removing duplicates
Elements 566 890
1) Time: 2 sec

DateTime start = DateTime.Now;
List<MyObject> removed = retValTmp.Distinct(new Comparer()).ToList();
double sec = Math.Round((DateTime.Now - start).TotalSeconds, 3);

2)时间:1.5秒

start = DateTime.Now;
List<MyObject> retList = new List<MyObject>();
HashSet<MyObject> removed2 = new HashSet<MyObject>(new Comparer());
foreach (var item in retValTmp)
{
    if (!removed2.Contains(item))
    {
        removed2.Add(item);
        retList.Add(item);
    }
}                
double sec2 = Math.Round((DateTime.Now - start).TotalSeconds, 3);

4)我也尝试过这种方式:

4) Also I tried out this way:

start = DateTime.Now;

var removed3 = retValTmp.Select(myObj => new { myObj.IdLoc, myObj.IdMet, myObj.Ser, myObj.Value, myObj.IdDataType, myObj.Time }).Distinct().ToList();

double sec3 = Math.Round((DateTime.Now - start).TotalSeconds, 3);

时间:0.35秒
但是返回的列表不在我的课程中，为什么列表1和2中的元素数与列表3中的元素数不同?

Time: 0.35 sec
but returned list is not in my class, and why the number of elements in the list of 1 and 2 is different than the list of 3?

UPDATE2

public int GetDataHashCode(MyObject obj)
{
    long idLoc = obj.IdLoc.HasValue ? obj.IdLoc.Value : 0;
    long idMet = obj.IdMet.HasValue ? obj.IdMet.Value : 0;
    long ser = obj.SerHasValue ? obj.Ser.Value : 0;
    int valueHash = 0;
    if (obj.Value != null)
        valueHash = obj.Value.GetHashCode();
    else
        valueHash = valueHash.GetHashCode();

    return (idLoc.GetHashCode() + idMet.GetHashCode() + ser.GetHashCode() + valueHash  + obj.IdDataType.GetHashCode() + obj.Time.GetHashCode()).GetHashCode();
}

使用:

foreach (MyObject daItem in retValTmp)
{
    int key = GetDataHashCode(daItem);
    if (!clearDict.ContainsKey(key))
        clearDict.Add(key, daItem);
} 

元素:750 000
时间: 0.23秒！

Element: 750 000
Time: 0.23 sec!

推荐答案

如果您要寻找的是速度，并且不介意占用一些内存，那么我建议您使用HashSet有兴趣进行自定义比较的用户，则可以制作一个IEqualityComparer<T>，如下所示:

If what you are looking for is speed, and don't mind using up some memory then I would recommend that you use a HashSet, if you are interested in doing some custom comparison, then you can make an IEqualityComparer<T>, something like this:

var original = new ArrayList(); // whatever your original collection is 
var unique = new HasSet<YourClass>(new MyCustomEqualityComparer());

foreach(var item in original)
{
    if(!unique.Contains(item))
        unique.Add(item);
}

return unique;

这里的问题是，您最终可能会吞噬原始内存的两倍.

the issue here is that you may end up gobbling up twice the original memory.

我做了一些额外的研究，我认为您只需做以下即可达到您想要的目标:

I made some extra research and I think you can achieve just what you want by simply doing:

var original // your original data
var unique = new HashSet<YourClass>(origin, new CustomEqualityComparer());

应该注意删除重复的数据，因为HashSet不允许重复.我建议您还看看关于GetHasCode实施指南的问题.

that should take care of removing duplicated data as no duplication is allowed in a HashSet. I'd recommend that you also take a look at this question about GetHasCode implementation guidelines.

如果您想进一步了解HashSet类，请遵循以下链接:

If you want to know some more about the HashSet class follow these links:

关于HashSet
关于IEqualityComparer构造函数
IEqualityComparer文档

About HashSet
About IEqualityComparer constructor
IEqualityComparer documentation

希望这会有所帮助

这篇关于从对象列表中删除重复项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！