Mathematica 内存不足 [英] Mathematica running out of memory

问题描述

我正在尝试运行以下程序，该程序计算系数仅为 +1 或 -1 的 d 次多项式的根，然后将其存储到文件中.

I'm trying to run the following program, which calculates roots of polynomials of degree up to d with coefficients only +1 or -1, and then store it into files.

d = 20; n = 18000; 
f[z_, i_] := Sum[(2 Mod[Floor[(i - 1)/2^k], 2] - 1) z^(d - k), {k, 0, d}];

这里 f[z,i] 给出了一个带正负号的 z 多项式，以二进制计数.假设 d=2，我们会有

Here f[z,i] gives a polynomial in z with plus or minus signs counting in binary. Say d=2, we would have

f[z,1] = -z² - z - 1
f[z,2] = -z² - z + 1
f[z,3] = -z² + z - 1
f[z,4] = -z² + z + 1

f[z,1] = -z² - z - 1
f[z,2] = -z² - z + 1
f[z,3] = -z² + z - 1
f[z,4] = -z² + z + 1

DistributeDefinitions[d, n, f]

ParallelDo[ 
            Do[ 
                     root = N[Root[f[z, i], j]];
                     {a, b} = Round[n ({Re[root], Im[root]}/1.5 + 1)/2];
            {i, 1, 2^d}],
{j, 1, d}]

我意识到阅读这篇文章可能不太有趣，但无论如何它都相对较短.我会尽量减少到相关部分，但在这里我真的不知道问题是什么.我正在计算 f[z,i] 的所有根，然后将它们四舍五入以使它们对应于 n x n 网格中的一个点，并将该数据保存在各种文件中.

I realise reading this probably isn't too enjoyable, but it's relatively short anyway. I would've tried to cut down to the relevant parts, but here I really have no clue what the trouble is. I'm calculating all roots of f[z,i], and then just round them to make them correspond to a point in a n by n grid, and save that data in various files.

出于某种原因，Mathematica 中的内存使用量逐渐增加，直到填满所有内存(在本机上为 6 GB)；然后计算继续非常缓慢；这是为什么?

For some reason, the memory usage in Mathematica creeps up until it fills all the memory (6 GB on this machine); then the computation continues extremely slowly; why is this?

我不确定是什么用完了这里的内存 - 我唯一的猜测是文件流用完了内存，但事实并非如此:我尝试将数据附加到 2GB 文件，但没有明显的内存使用情况.Mathematica 似乎完全没有理由在这里使用大量内存.

I am not sure what is using up the memory here - my only guess was the stream of files used up memory, but that's not the case: I tried appending data to 2GB files and there was no noticeable memory usage for that. There seems to be absolutely no reason for Mathematica to be using large amounts of memory here.

对于较小的 d 值(例如 15)，行为如下:我有 4 个内核正在运行.由于它们都运行在 ParallelDo 循环中(每个循环一次执行一个 j 值)，内存使用量会增加，直到它们都完成一次该循环.然后下一次他们通过该循环时，内存使用量根本不会增加.计算最终完成，一切正常.

For small values of d (15 for example), the behaviour is the following: I have 4 kernels running. As they all run through the ParallelDo loop (each doing a value of j at a time), the memory use increases, until they all finish going through that loop once. Then the next times they go through that loop, the memory use does not increase at all. The calculation eventually finishes and everything is fine.

另外，非常重要的是，一旦计算停止，内存使用量不会下降.如果我开始另一个计算，则会发生以下情况:

Also, quite importantly, once the calculation stops, the memory use does not go back down. If I start another calculation, the following happens:

-如果之前的计算在内存使用量仍在增加时停止，则它会继续增加(可能需要一段时间才能再次开始增加，基本上是在计算中达到相同点).

-If the previous calculation stopped when memory use was still increasing, it continues to increase (it might take a while to start increasing again, basically to get to the same point in the computation).

-如果之前的计算在内存使用没有增加时停止，则不会进一步增加.

-If the previous calculation stopped when memory use was not increasing, it does not increase further.

问题似乎来自 f 的相对复杂性 - 将其更改为更简单的多项式似乎可以解决问题.我认为问题可能是 Mathematica 记住了 f[z,i] 的特定 i 值，但设置了 f[z,i] :=.就在计算了 f[z,i] 的根之后，抱怨首先分配不存在，并且仍然使用内存.

The issue seems to come from the relative complexity of f - changing it into some easier polynomial seems to fix the issue. I thought the problem might be that Mathematica remembers f[z,i] for specific values of i, but setting f[z,i] :=. just after calculating a root of f[z,i] complains that the assignment did not exist in the first place, and the memory is still used.

这真的很令人费解，因为 f 是我能想象到的唯一占用内存的东西，但是在内部 Do 循环中定义 f 并在每次计算根后清除它并不能解决问题.

It's quite puzzling really, as f is the only remaining thing I can imagine taking up memory, but defining f in the inner Do loop and clearing it each time after a root is calculated does not solve the problem.

推荐答案

哎呀，这是一个令人讨厌的方案.

Ouch, this is a nasty one.

现在的情况是 N 将缓存结果，以便在您再次需要时加快计算速度.有时这绝对是您想要的，但有时它只是打破了世界.幸运的是，您确实有一些选择.一种是使用 ClearSystemCache 命令，它不会正是它在罐头上所说的.在我运行您的非并行循环一段时间后(在感到无聊并中止计算之前)，MemoryInUse 报告了大约 160 MiB 正在使用中.使用 ClearSystemCache 将其降低到大约 14 MiB.

What's going on is that N will do caching of results in order to speed up future calculations if you need them again. Sometimes this is absolutely what you want, but sometimes it just breaks the world. Fortunately, you do have some options. One is to use the ClearSystemCache command, which does just what it said on the tin. After I ran your un-parallelized loop for a little while (before getting bored and aborting the calculation), MemoryInUse reported ~160 MiB in use. Using ClearSystemCache got that down to about 14 MiB.

您应该考虑做的一件事是使用 ClearSystemCacherel="noreferrer">SetSystemOptions 更改缓存行为.您应该查看 SystemOptions["CacheOptions"] 以了解可能的情况.

One thing you should look at doing, instead of calling ClearSystemCache programmatically, is to use SetSystemOptions to change the caching behavior. You should take a look at SystemOptions["CacheOptions"] to see what the possibilities are.

对于更复杂的表达式，缓存会导致更大的问题，这并不奇怪.它必须将这些表达式的副本藏在某处，而更复杂的表达式需要更多的内存.

It's not terribly surprising that the caching causes a bigger problem for more complex expressions. It's got to be stashing copies of those expressions somewhere, and more complex expressions require more memory.

这篇关于Mathematica 内存不足的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！