如何避免在高内存使用情况的应用程序运行内存？ C / C ++ [英] How to avoid running out of memory in high memory usage application? C / C++

问题描述

我写了一个转换器，采用OpenStreetMap的XML文件，并将其转换为通常是原始大小的大约10％的二进制运行时呈现格式。输入文件大小通常3GB大。输入文件不会被加载到内存中的所有在一次，但由于流点和多边形收集，那么BSP是在它们上面运行和文件输出。最近对大文件它运行的内存和死亡（一个在问题有1400万个100万和多边形）。通常，我的计划是用约1GB RAM到1.2 GB时，发生这种情况。我试着从2增加虚拟内存到8GB（XP系统），但这种变化做出无影响。此外，由于此code是开源的，我想不管有它的工作可用RAM的（尽管速度较慢），它运行在Windows，Linux和Mac。

I have written a converter that takes openstreetmap xml files and converts them to a binary runtime rendering format that is typically about 10% of the original size. Input file sizes are typically 3gb and larger. The input files are not loaded into memory all at once, but streamed as points and polys are collected, then a bsp is run on them and the file is output. Recently on larger files it runs out of memory and dies (the one in question has 14million points and 1million polygons). Typically my program is using about 1gb to 1.2 gb of ram when this happens. I've tried increasing virtual memory from 2 to 8gb (on XP) but this change made no effect. Also, since this code is open-source I would like to have it work regardless of the available ram (albeit slower), it runs on Windows, Linux and Mac.

我可以使用哪些技术来避免它运行内存不足？在更小的子集的处理的数据，然后合并的最终结​​果？使用处理我自己的虚拟内存类型？任何其他的想法？

What techniques can I use to avoid having it run out of memory? Processing the data in smaller sub-sets and then merging the final results? Using my own virtual memory type of handler? Any other ideas?

推荐答案

首先，在32位系统上，你总是会被限制在4 GB的内存，无论是页面文件的设置。 （而这些，只有2GB将提供给您的Windows进程。在Linux中，您通常有大约3GB可用）

First, on a 32-bit system, you will always be limited to 4 GB of memory, no matter pagefile settings. (And of those, only 2GB will be available to your process on Windows. On Linux, you'll typically have around 3GB available)

因此​​，第一个显而易见的解决方案是切换到64位操作系统和编译为64位应用程序。这就给了你一个巨大的虚拟内存空间使用，操作系统将进出必要的页面文件，以保持事情的工作交换数据。

So the first obvious solution is to switch to a 64-bit OS, and compile your application for 64-bit. That gives you a huge virtual memory space to use, and the OS will swap data in and out of the pagefile as necessary to keep things working.

二，同时可以帮助分配的内存较小的块。它往往更容易找到的可用内存256MB 4块以上的1GB数据块。

Second, allocating smaller chunks of memory at a time may help. It's often easier to find 4 256MB chunks of free memory than one 1GB chunk.

三，分裂的问题。不要一次处理整个数据集，但尝试一次加载和处理只有一小部分。

Third, split up the problem. Don't process the entire dataset at once, but try to load and process only a small section at a time.

这篇关于如何避免在高内存使用情况的应用程序运行内存？ C / C ++的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！