Matlab:调用函数后，空闲内存丢失 [英] Matlab: Free memory is lost after calling a function
问题描述
我在Matlab中的内存管理遇到了一些麻烦.最后，它导致可用内存不足和错误.我试图指出问题所在，并发现了一个有趣的功能":以某种方式我在Matlab中释放了可用内存.
我执行以下操作:
1)启动Matlab
2)输入"memory"，我得到:最大可能的数组:1293 mb，可用于所有数组的内存:1456 mb
3)我将调用一个函数.该函数相当长，因此很难将其粘贴到此处.但基本上它加载5ca. 300mb的Mat文件(顺序)，选择一些值并将其返回.返回的矩阵约为. 1,2mb(4650x35两倍)
4)我清除工作区中的所有变量(全部清除")
5)输入"memory"，我得到:最大可能的数组:759 mb，所有数组可用的内存:1029 mb
如果我重复步骤3)到5)，则内存号是恒定的.
那么这里出什么问题了?我该在哪里释放400mb的可用空间? Matlab使用的内存恒定在330mb左右.
有人有什么主意吗?还是这是完全自然的事情，但是我想念吗?
谢谢
托马斯
此丢失"内存的很大一部分可能是由于内存碎片.在会话过程中，随着Matlab分配和释放数组，内存被分解成较小的区域，并且在Matlab和底层C级别的内存管理器中，一些内存丢失了. Matlab不会将开销算作已使用"，因为它并未用于保存M代码数组值. Matlab加载其他M文件和库，分配内部缓冲区或结构，或者扩展Matlab嵌入式JVM中的Java堆，也可能会消耗一些内存.这是正常的.完成一些工作后，Matlab将没有新会话中的可用内存.
AFAIK，一旦发生低级碎片，除了重新启动Matlab之外，您无法采取任何其他措施来消除碎片.分配大量的小阵列可以加速碎片化.如果您使用大型单元格或大量对象，则有时会发生这种情况.因此，如果遇到问题，可能需要通过将工作分成较小的块，减少单元使用率等来减少函数中的峰值内存使用率.如果MAT文件中有很大的cellstr数组，请将其转换为char.分配的高水位线"是支配碎片的因素，因此，如果可以将数据集分成较小的块，则可以将其装入较少的内存中.
在您的函数内部，请先从一个MAT文件中清除尽可能多的内容，然后再继续进行下一个操作.隐式执行此操作的一种方法是，如果每个文件处理当前位于主函数中的循环中，则将其移至子函数中.
为帮助调试，请执行"dbstop if all error"(如果所有错误，dbstop)，该错误将由OOM触发.从那里，您可以使用whos和调试器来找出耗尽内存时占用空间的位置.这可能会揭示需要清除的临时变量，或者建议将工作分块的方法.
如果您想尝试查看碎片的外观以及它如何影响memory()的输出，则此函数将创建一些碎片.
function fragmem(nbytes, chunksize)
%FRAGMEM Fragment the Matlab session's memory
if nargin < 2; chunksize = 1*2^10; end
nbytes = nbytes - rem(nbytes, chunksize);
nsteps = 100; % to make initial input relatively small
c = cell([1 nsteps]);
stepsize = nbytes / nsteps;
chunksperstep = ceil(stepsize / chunksize);
fprintf('Fragmenting %d MB memory into %d KB chunks (%d steps of %d chunks)\n',...
round(nbytes/2^20), round(chunksize/2^10), nsteps, chunksperstep);
x = zeros([1 chunksperstep * chunksize], 'uint8');
colsizes = repmat(chunksize, [1 chunksperstep]);
for i = 1:nsteps
c{i} = mat2cell(x, 1, colsizes);
end
在我的计算机上以1KB的块大小压缩300 MB会在win32计算机上再现大约所见大小的损失".
>> memory
Maximum possible array: 1384 MB (1.451e+009 bytes) *
Memory available for all arrays: 1552 MB (1.627e+009 bytes) **
Memory used by MATLAB: 235 MB (2.463e+008 bytes)
Physical Memory (RAM): 3311 MB (3.472e+009 bytes)
>> fragmem(300*2^20)
Fragmenting 300 MB memory into 1 KB chunks (100 steps of 3072 chunks)
>> memory
Maximum possible array: 1009 MB (1.059e+009 bytes) *
Memory available for all arrays: 1175 MB (1.232e+009 bytes) **
Memory used by MATLAB: 257 MB (2.691e+008 bytes)
Physical Memory (RAM): 3311 MB (3.472e+009 bytes)
>>
I have some troubles with memory management in Matlab. Finally it leads to not enough free memory and an error.I tried to pinpoint the problem and found one interesting "feature": Somehow I loose free Memory in Matlab.
I do the following:
1) Start Matlab
2) typing "memory" I get: Maximum possible array: 1293 mb, Memory available for all arrays: 1456 mb
3) I'll call a function. The function is rather long, so it's hard to paste it here. But basically it loads 5 ca. 300mb mat files (sequentially), picks some few values and returns them. The returned matrix is ca. 1,2mb (4650x35 double)
4) I clear all variables in workspace ("clear all")
5) typing "memory" I get: Maximum possible array: 759 mb, Memory available for all arrays: 1029 mb
If I repeat steps 3) to 5) the memory numbers are constant.
So what is wrong here? Where do I loose the 400mb of free space? The memory used by Matlab is constant at around 330mb.
Does anyone have some ideas what is wrong here? Or is this something totally natural, but I miss it??
Thanks
Thomas
Ps: I use Matlab 2010a and Win 7 pro 32bit.
A good part of this "lost" memory is probably due to memory fragmentation. As Matlab allocates and frees arrays over the course of a session, the memory gets broken up into smaller areas, and some is lost to overhead in the memory manager, at both the Matlab and the underlying C levels. The overhead is not counted as "used" by Matlab because it's not being used to hold M-code array values. Some memory may also be consumed by Matlab loading additional M-files and libraries, allocating internal buffers or structures, or by expansion of the Java heap in Matlab's embedded JVM. This is normal. After doing some work, Matlab won't have as much memory available as it did in a fresh session.
AFAIK, once low-level fragmentation occurs, there's nothing you can do to eliminate it aside from restarting Matlab. Allocating lots of small arrays can accelerate fragmentation. This sometimes happens if you use large cellstrs or large arrays of objects. So if you are having problems, you may need to reduce your peak memory usage in the function by breaking the work in to smaller chunks, reducing cell usage, and so on. And if you have big cellstr arrays in the MAT files, convert them to char. The "high water mark" of allocation is what governs fragmentation, so if you can break your data set in to smaller chunks, you can fit it in less memory.
Inside your function, clear as much as you can from one MAT file before moving on to the next. One way to do this implicitly is to move the per-file processing into a subfunction if it's currently sitting in a loop in your main function.
To help debug, do a "dbstop if all error", which will get triggered by the OOM. From there, you can use whos and the debugger to find out where the space is being taken up when you exhaust memory. That might reveal temp variables that need to be cleaned up, or suggest ways of chunking the work.
If you'd like to experiment to see what fragmentation looks like and how it affects memory()'s output, here's a function that will just create some fragmentation.
function fragmem(nbytes, chunksize)
%FRAGMEM Fragment the Matlab session's memory
if nargin < 2; chunksize = 1*2^10; end
nbytes = nbytes - rem(nbytes, chunksize);
nsteps = 100; % to make initial input relatively small
c = cell([1 nsteps]);
stepsize = nbytes / nsteps;
chunksperstep = ceil(stepsize / chunksize);
fprintf('Fragmenting %d MB memory into %d KB chunks (%d steps of %d chunks)\n',...
round(nbytes/2^20), round(chunksize/2^10), nsteps, chunksperstep);
x = zeros([1 chunksperstep * chunksize], 'uint8');
colsizes = repmat(chunksize, [1 chunksperstep]);
for i = 1:nsteps
c{i} = mat2cell(x, 1, colsizes);
end
Fragging 300 MB in 1KB chunks on my machine reproduces a "loss" on my win32 machine about the size you're seeing.
>> memory
Maximum possible array: 1384 MB (1.451e+009 bytes) *
Memory available for all arrays: 1552 MB (1.627e+009 bytes) **
Memory used by MATLAB: 235 MB (2.463e+008 bytes)
Physical Memory (RAM): 3311 MB (3.472e+009 bytes)
>> fragmem(300*2^20)
Fragmenting 300 MB memory into 1 KB chunks (100 steps of 3072 chunks)
>> memory
Maximum possible array: 1009 MB (1.059e+009 bytes) *
Memory available for all arrays: 1175 MB (1.232e+009 bytes) **
Memory used by MATLAB: 257 MB (2.691e+008 bytes)
Physical Memory (RAM): 3311 MB (3.472e+009 bytes)
>>
这篇关于Matlab:调用函数后，空闲内存丢失的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
