仅当有足够的可用内存时，才进行向量push_back [英] Vector push_back only if enough memory is available

问题描述

我目前正在使用vector类动态构建处理大量内存的代码.

I am currently building a code that is dealing with massive amounts of memory using the vector class, dynamically.

代码使用push_back构建向量，需要注意的是向量是二维的，代表数据矩阵，这一点很重要.根据情况，此矩阵可能很小，也可能变得异常大.

The code is building up the vector with push_back, where it is important to notice that the vector is 2 dimensional, representing the data matrix. Depending on circumstances, this matrix can be small, or become exceptionally large.

例如，数据矩阵可以有几行，每行有1000列，或者它可以得到1000行，且列数相同，并且充满了双重数据类型.显然，这很容易成为问题，因为1000x1000x8 = 8000000字节，因此表示8 MB的内存.但是，列数增加10倍，行数增加10倍又如何呢? (这很容易在我的代码中发生).

For instance, data matrix can have few rows, with 1000 columns in every row, or it can get 1000 rows with the same amount of columns, full of double data types. Obviously, this can very easily become a problem, because 1000x1000x8 = 8 000 000 bytes, thus representing 8 MB in memory. But how about 10 times more columns and 10 times more rows? (which can easily happen in my code).

我正在通过将数据矩阵写入HDD来解决此问题，但是这种方法相当慢，因为我没有充分利用RAM.

I am solving this by writing the data matrix into HDD, however this approach is rather slow, because I am not using RAM to the fullest.

我的问题: 只有在有足够的内存可以分配的情况下，才能使用push_back构建由vector< vector<double> >表示的矩阵.

My question: How can I build this matrix represented by vector< vector<double> > using push_back, but only if there is enough memory that can be allocated.

如果内存不足，我将继续将数据导出到HDD到文件中，释放分配的内存并重新开始循环.我不知道如何检查每次执行的push_back内存是否可用.

If the amount of memory is not sufficient, I will continue by exporting the data to HDD into a file, freeing the allocated memory and starting the cycle over. What I don't know is how to check if memory is available with every push_back executed.

修改: 我应该已经注意到我正在使用运行Ubuntu的64位计算机.我不太确定如何以及如何运行OS分页，但是我实际上所做的是在存在电场和磁场的情况下对粒子进行数值计算.可能有1亿个粒子以1000多个时间步长移动，这是大量的GB数据.但是，有时我只运行了数十万个粒子就可以毫无问题地装入RAM，从而加快了计算过程.我正在尝试创建某种通用的解决方案，该解决方案将检查是否有足够的RAM用于其他计算，如果没有，则将其移动到文件中.这些粒子可以添加到系统中或从系统中流出，因此基本上我不知道在任何给定时间矩阵的大小.这就是为什么我需要足够了，将这些数据移出此处，以便我们可以重新开始"的方法.

I should have noticed that I am using 64 bit machine that runs Ubuntu. I am not quite sure how and if OS paging is running, however what I am actually doing is numerical computations of particles in presence of electric and magnetic fields. There can be 100 million of particles that are moving in over 1000 time steps, which is lot of GB of data. However, sometimes I am running just few hundred thousands of particles for tests that fit into RAM without a problem, speeding the computation process. I am trying to create somewhat generalized solution that will check if there is enough RAM for another computation, and if not then to move them into a file. Those particles can add into the system or flow out from it, so basically I have no idea of how large the matrix will be at any given time. That is why I need the "okay that's enough, move those data out of here so I we can start over" method.

推荐答案

我将代码中的数据推入磁盘"的几乎所有替代方法都比这更好.

Almost ALL alternatives to "I will push data to disk in my code" are better than that.

这是因为OS本身(如果我们说的是相当现代的OS，例如Windows NT系列和Unix的大多数变体，包括Linux和MacOS X)具有处理虚拟内存并交换到磁盘的能力，它将这样做比您想出的要聪明得多.

That's because the OS itself (if we're talking reasonably modern OS's such as Windows NT family and most variants of Unix, including Linux and MacOS X) has the ability to deal with virtual memory and swapping to disk, and it will do so in a more clever way than you are likely to come up with.

进一步(根据Tony D的评论)，与手动读取/写入文件相比，使用内存映射文件"是一种更好的方法-这不适用于std::vector或其他标准集合，但可能与手动处理应用程序中的文件读写相比，这是一个更好的选择-您只需说这是文件，请给我一个指向代表该文件的内存的指针"，然后就可以像使用该文件一样加载该指针进入记忆.操作系统将负责管理在任何给定时间文件的哪些部分实际实际存在于内存中，类似于如果您分配的内存多于系统中存在的内存，则换入和换出.

Further (as per Tony D's comment), using "memory mapped file" is a better method than manually reading/writing to a file - this won't work immediately with a std::vector or other standard collections, but is probably a better choice than manually dealing with reading/writing files in your application - you simply say "Here's a file, please give me a pointer to a piece of memory that represents that file", and you use that pointer as if the file was loaded into memory. The OS will take care of managing which parts of the file is ACTUALLY physically present in memory at any given time, similar to swapping in and out if you allocate more memory than there is present in the system.

但是，这当然有局限性(适用于为应用分配更多的可用RAM和内存映射文件解决方案".如果您使用的是32位计算机(或32位计算机)操作系统或32位应用程序)，您的进程可使用的最大内存量将介于2GB和4GB之间-确切的限制取决于操作系统(带有32位应用程序的64位操作系统可能会为您提供近4GB的存储空间，常规的32位Windows安装程序总共需要2GB的空间.因此，如果您的阵列足够大，则地址中就不会有足够的位"来跟踪它了.或转到64位操作系统和应用程序(此处自然需要64位处理器)，在这种情况下，内存大小的限制为128或256TB(如果我的智力算术有效-65536 * 4GB)，总计-可能几乎是几乎每个人都拥有的磁盘空间，不用担心RAM.

However, there are of course limits to this (applies to both the "allocate more than there is RAM available for your app and the memory mapped file solution). If you are using a 32-bit machine (or 32-bit OS or 32-bit application), the maximum amount of memory available to your process will be somewhere between 2GB and 4GB - exactly what the limit is depends on the OS (64-bit OS with 32-bit app may give you nearly 4GB, regular setup of 32-bit Windows gives about 2GB total). So if your array gets big enough, there simply won't be "enough bits" in the address to keep track of it. At which point you need to split the work in some way. Or go to 64-bit OS and application (and naturally a 64-bit processor is needed here), in which case the limit to memory size goes to 128 or 256TB (if my mental aithmetics works - 65536 * 4GB) in total - which is probably more than nearly everyone has as disk-space, never mind RAM.

根据您提供的数据进行一些数学运算:每个粒子具有X，Y，Z位置，速度和其他两个属性"，在double中将占用6 * 8 = 48个字节，并且6 * 4 = 24个字节作为float.

Doing some math based on the data you've given: with each particle having X, Y, Z position, a velocity, and "two other properties" would take up 6 * 8 = 48 bytes in double, and 6 * 4 = 24 bytes as float.

乘以100M，我们将获得4.8GB的一组数据.乘以1000个时间步长，即可产生4.8TB的数据.即使您有非常大的内存，这也是一个巨大的数目.使用内存映射文件实际上无法一次将所有这些数据保存在内存中.如果您的机器上有足够的内存(16GB左右)，则一次保存两套内存可能会起作用.但是您仍在生成大量需要在某个时间存储的数据，这很可能会花费大部分时间.对于合理的现代(单个)硬盘，大约50-100MB/s是合理的期望.可以通过某些RAID配置进行改进，但即使是RAID配置，也可以达到每秒数百兆字节，而每秒不超过千兆字节.因此，以100MB/s的速度存储1 TB(1000GB)将会花费10000s，或大约三个小时. 4.8 TB需15小时.只是存储数据而无需计算[尽管这可能只是很小的一部分].即使我们将数据集除以10，我们仍然有一个多小时，再除以50，则我们在分钟数范围内下降.

Multiply by 100M and we get 4.8GB for one set of data. Times 1000 timesteps, makes 4.8TB of data. That's a huge amount, even if you have a really large amount of memory. Using mememory mapped files is not really going to work to hold all this data in memory at once. If you have a machine with a decent amount of memory (16GB or so), keeping TWO sets in memory at a time would likely work. But you're still producing a lot of data that needs to be stored at some point, which will most likely take most of the time. For a reasonably modern (single) hard disk, somewhere around 50-100MB/s would be a reasonable expectation. It can be improved by certain RAID configurations, but even them, it's hundreds of megabytes per second, not many gigabytes per second. So, to store 1 TB (1000GB) at 100MB/s would take 10000s, or roughly three hours. 15 hours for 4.8TB. That's JUST to store the data, no calculation [although that is probably a minimal part]. Even if we divide the data set by 10, we have more than an hour, and divide by 50 and we're down in the minutes range.

无论使用哪种方法，至少可以说，存储和检索如此大的数据集都是很耗时的.内存映射文件在许多方面都是最糟糕的"，因为它在此过程中复制的数据要少得多.但是，仍然是磁盘速度"将成为计算速度的主要因素.

No matter WHAT method you use, storing and retrieving such large data-sets is time-consuming to say the least. Memory mapped files is the "least bad" in many ways, because it copies the data a little less in the process. But it's still "disk speed" that will be the dominant factor for your calculation speed.

这篇关于仅当有足够的可用内存时，才进行向量push_back的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！