良好的C ++数组类与大型数据阵列在一个快速和内存高效的方式处理？ [英] Good C++ array class for dealing with large arrays of data in a fast and memory efficient way?
问题描述
从<一个继href=\"http://stackoverflow.com/questions/2469738/unusual-heap-size-limitations-in-vs2003-c\">$p$pvious有关堆的使用限制问题，我正在寻找一个很好的标准C ++类与大数据阵列的方式，既高效利用内存和速度高效处理。我一直在分配使用单一的malloc / HealAlloc数组，但使用不同的电话多次改掉后，继续下跌堆碎片的犯规。所以结论我来，除了移植到64位，是使用一种机制，让我有一个大阵跨越多个更小的内存碎片。我不希望每个元素的页头，因为这是非常低效的内存，因此该计划是写一个覆盖[]操作符类，并选择基于该指数的适当元素。是否已有一个体面的类在那里做到这一点，还是我最好滚动我自己？
Following on from a previous question relating to heap usage restrictions, I'm looking for a good standard C++ class for dealing with big arrays of data in a way that is both memory efficient and speed efficient. I had been allocating the array using a single malloc/HealAlloc but after multiple trys using various calls, keep falling foul of heap fragmentation. So the conclusion I've come to, other than porting to 64 bit, is to use a mechanism that allows me to have a large array spanning multiple smaller memory fragments. I don't want an alloc per element as that is very memory inefficient, so the plan is to write a class that overrides the [] operator and select an appropriate element based on the index. Is there already a decent class out there to do this, or am I better off rolling my own?
从我的理解，有的谷歌搜索，32位Windows进程应理论上能够地址可达2GB。现在假设我已经安装了2GB，以及其他各种流程和服务会占用400MB左右，多少可用内存，你认为我的计划可以合理预期从堆中获得？
From my understanding, and some googling, a 32 bit Windows process should theoretically be able address up to 2GB. Now assuming I've 2GB installed, and various other processes and services are hogging about 400MB, how much usable memory do you think my program can reasonably expect to get from the heap?
目前我使用了Visual C ++各种口味。
I'm currently using various flavours of Visual C++.
修改按Poita的帖子，我已经尝试了的std :: deque的，使用下面的测试上VS2008;
Edit As per Poita's post, I've tried a std::deque, using the following test on VS2008;
#include <deque>
using namespace std;
struct V
{
double data[11];
};
struct T
{
long data[8];
};
void dequeTest()
{
deque<V> VQ;
deque<T> TQ;
V defV;
T defT;
VQ.resize(4000000,defV);
TQ.resize(8000000,defT);
}
对于上述数据的总内存出来为608MB，被我用直的malloc或HeapAlloc，并采取＆LT; 1秒。双端队列重新调整了950MB原本，然后慢慢地开始下降了。 15分钟后，dequeTest（）完成，使用的内存为显示的过程这可能是更多地与运行时间只需6MB。我也尝试填充使用各种推送选项的双端队列，但表现得如此糟糕，我不得不提前爆发。我可能提供比defualt更好的分配以获得更好的响应，但在它的面前deque的是不是这个职位的类。请注意，这也可能涉及到MS VS2008实现双端队列的，因为似乎在这个类中，这是非常依赖于实现被很多，当谈到性能。
The total memory for the above data comes out at 608MB, were I to use straight malloc or HeapAlloc, and takes < 1 second. The deque resizes took 950MB originally, and then slowly started dropping back. 15 minutes later, dequeTest() finished, using just 6MB of memory showing for the process which probably was more to do with the run-times. I also tried populating the deque using various push options, but performance was so bad, I had to break out early. I could possibly provide a better allocator than the defualt to get a much better response, but on the face of it deque is not the class for this job. Note this could also relate to the MS VS2008 implementation of deque, as there seems to be alot in this class that is very implementation dependant when it comes to performance.
时间写我自己的大数组类，我想。
Time to write my own big array class, I reckon.
第二个编辑：分配少量产生1.875GB立即用以下;
Second Allocating smaller amounts yielded 1.875GB immediately using the following;
#define TenMB 1024*1024*10
void SmallerAllocs()
{
size_t Total = 0;
LPVOID p[200];
for (int i = 0; i < 200; i++)
{
p[i] = malloc(TenMB);
if (p[i])
Total += TenMB; else
break;
}
CString Msg;
Msg.Format("Allocated %0.3lfGB",Total/(1024.0*1024.0*1024.0));
AfxMessageBox(Msg,MB_OK);
}
最后修改我已经决定接受Poita的职位和各种意见之后，也不是因为我会直接使用双端队列类，但多为阵列卡概念在甲板随后的评论。这应该是简单的用O（1）随机元素访问来实现，基于固定数量的每个块的元素，这正是我需要的。感谢所有的反馈！
Final edit I have decided to accept Poita's post and the various comments following it, not because I'll be using the deque class directly, but more for the array as a deck of cards notion in the comments that followed. This should be straightforward to implement with O(1) random element access, based on a fixed number of elements per block, which is what i need. Thanks to all for the feedback!
推荐答案
您是否尝试过使用的std :: deque的？不同于的std ::矢量，它使用一个巨大的堆分配，双端通常分配一小块一小块，但仍然提供通过分期常量时间的索引运算符[] 。
Have you tried using an std::deque? Unlike a std::vector, which uses one huge heap allocation, deque usually allocates in small chunks, but still provides amortised constant time indexing via operator[].
这篇关于良好的C ++数组类与大型数据阵列在一个快速和内存高效的方式处理？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
