如何避免OrderBy-内存使用问题 [英] How to avoid OrderBy - memory usage problems

问题描述

假设我们有大量的点列表. List< Point>pointList (已经存储在内存中)，其中每个 Point 都包含X，Y和Z坐标.

Let's assume we have a large list of points List<Point> pointList (already stored in memory) where each Point contains X, Y, and Z coordinate.

现在，我想选择例如存储在 pointList 中的所有点中具有最大Z值的N％点.现在我正在那样做:

Now, I would like to select for example N% of points with biggest Z-values of all points stored in pointList. Right now I'm doing it like that:

N = 0.05; // selecting only 5% of points
double cutoffValue = pointList
    .OrderBy(p=> p.Z) // First bottleneck - creates sorted copy of all data
    .ElementAt((int) pointList.Count * (1 - N)).Z;

List<Point> selectedPoints = pointList.Where(p => p.Z >= cutoffValue).ToList();

但是我这里有两个内存使用瓶颈:首先是在OrderBy(更重要)期间，其次是在选择点时(这不太重要，因为我们通常只选择少量的点).

But I have here two memory usage bottlenecks: first during OrderBy (more important) and second during selecting the points (this is less important, because we usually want to select only small amount of points).

有什么方法可以用内存更少的东西替换OrderBy(或者找到该截止点的其他方法)?

Is there any way of replacing OrderBy (or maybe other way of finding this cutoff point) with something that uses less memory?

这个问题非常重要，因为LINQ复制了整个数据集，而对于我正在处理的大文件，有时会达到数百MB.

The problem is quite important, because LINQ copies the whole dataset and for big files I'm processing it sometimes hits few hundreds of MBs.

推荐答案

您可以使用使用快速排序算法的 List.Sort 就地排序列表.但是，当然，您的原始列表将被排序，这可能不是您想要的...

You could sort the list in place, using List<T>.Sort, which uses the Quicksort algorithm. But of course, your original list would be sorted, which is perhaps not what you want...

pointList.Sort((a, b) => b.Z.CompareTo(a.Z));
var selectedPoints = pointList.Take((int)(pointList.Count * N)).ToList();

如果您不介意对原始列表进行排序，那么这可能是内存使用率和速度之间的最佳平衡

If you don't mind the original list being sorted, this is probably the best balance between memory usage and speed

这篇关于如何避免OrderBy-内存使用问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！