在长时间运行的应用程序中减少numpy内存占用 [英] Reducing numpy memory footprint in long-running application

问题描述

在我的应用程序中，生成一百个numpy数组（每个1000个复合元素）并填充数据。然后在许多迭代中，数组元素会一遍又一遍地被修改。初始生成后，系统监视器报告约50 Mb的RAM使用情况。虽然我没有生成任何新的数组，但占位面积每迭代增长大约40 Mb。

我学会了这里，垃圾收集器不处理numpy数组。所以我假设我生成的用于处理数据的临时数组无法正确收集。

它说，guppy.hpy（）。堆（）不帮助分析numpy，不幸的是， 。

如何确定问题的根源并理想地保持消耗在任何次数的迭代中保持不变？

我怀疑当分配数组元素时可能会产生副本

我可以手动处理临时numpy数组来协助垃圾回收吗？

可以手动处理临时numpy数组来协助垃圾回收吗？

[更新1]：示例代码

这段代码被称为数千次。每次，足迹都会增加。我看不出为什么，因为根据我的理解，它只是读取现有数组并操作其他现有数组。这些切片操作中的任何一个是否在意料之中？ （对不起，我可以简化它，但是我也可能隐藏我的错误。）

  for ts在np.arange（numTimeslots）中：
用于np.arange（numFreqChunks）中的fc：
 interfencep = np.sum（np.dot（np.dot（self.baseStations [bs] .cells [cell ] .CSI_OFDMA [：，：，fc，ts]，np.diag（cell.OFDMA_power [：，fc，ts]）），self.baseStations [bs] .cells [cell] .CSI_OFDMA [：，：，fc， （self.antennas）*（self.noisePower / numFreqChunks（））。 ）
 self.OFDMA_interferenceCovar [：，：，fc，ts] = noisep + interfencep 
 self.OFDMA_EC [：，：，fc，ts] =（np.dot（np.dot（self.OFDMA_CSI [：，：，fc，ts]，linalg.inv（noisep + interfencep）），self.OFDMA_CSI [：，：，fc，ts] .conj（）。T））
 eigs = linalg.eig self.OFDMA_EC [：，：，fc，ts]）[0] 
 self.OFDMA_SINR [：，fc，ts] = np.real（eigs）
  pre> 
 
 
 [更新2]：F或者那些好奇的人，这是移动网络模拟器的一部分。运行在virtualenv上，Python 2.7.3，Numpy 1.6.2，SciPy 0.11.0b1 
 
 
 [更新3]：通过评论并检查系统监视器，我可以识别作为罪魁祸首的'interferencep = ...'线。它分配了没有释放的重要内存。但为什么？通过使用系统监视器和代码检查/评论，我发现内存泄漏。为什么？
 它是由在不同文件中将一个numpy数组与一个空列表进行比较而引起的。我会在不同的地方挖掘漏洞，并投票删除这个问题，因为我觉得这个问题太具体以至于不能帮助其他人。 
 
 <更新1>：描述新问题问题的根源：
 为什么numpy数组与列表的比较会消耗太多内存？ 
 
In my application one hundred numpy arrays (1000 complex elements each) are generated and filled with data. Then over many iterations, the array elements are modified over and over again. After the initial generation, the system monitor reports around 50 Mb of RAM usage. Although I am not generating any new arrays, the footprint keeps growing by around 40 Mb per iteration.

I learned here, that the garbage collector does not handle numpy arrays. So I assume that some temporary arrays I am generating to manipulate data are not collected correctly.

Here it says that guppy.hpy().heap() does not help with profiling numpy, unfortunately.

How can I identify the source of the problem and ideally keep consumption constant over any number of iterations?

I suspect that I may be generating copies when assigning array elements as described here, which are then not garbage collected.

Can I manually dispose of temporary numpy arrays to assist garbage collection?

[Update 1]: Sample code

This bit of code is called thousands of times. Each time, the footprint increases. I cannot see why, because to my understanding it is only reading existing arrays and manipulating other existing arrays. Are any of these slicing operations doing something unintended? (Sorry for the line length. I can simplify it, but then I might be hiding my errors, too.)
for ts in np.arange(numTimeslots):
            for fc in np.arange(numFreqChunks):
                interfencep = np.sum( np.dot(np.dot(self.baseStations[bs].cells[cell].CSI_OFDMA[:,:,fc,ts] ,np.diag(cell.OFDMA_power[:,fc,ts])),self.baseStations[bs].cells[cell].CSI_OFDMA[:,:,fc,ts].conj().T) for bs in self.baseStations for cell in bs.cells if cell != self._cell) 
                noisep = np.eye(self.antennas) * (self.noisePower / numFreqChunks)
                self.OFDMA_interferenceCovar[:,:,fc,ts] = noisep + interfencep
                self.OFDMA_EC[:,:,fc,ts] = (np.dot(np.dot(self.OFDMA_CSI[:,:,fc,ts],linalg.inv(noisep+interfencep)),self.OFDMA_CSI[:,:,fc,ts].conj().T))
                eigs = linalg.eig(self.OFDMA_EC[:,:,fc,ts])[0]
                self.OFDMA_SINR[:,fc,ts] = np.real(eigs)
[Update 2]: For those curious, this is part of a mobile network simulator. Running on virtualenv, Python 2.7.3, Numpy 1.6.2, SciPy 0.11.0b1

[Update 3]: Via commenting it and checking the system monitor, I can identify the 'interferencep = ...'-line as a culprit. It allocated significant memory which is not freed. But why?
解决方案
Via using the system monitor and code inspection/commenting, I found the memory leak. It was caused by comparing a numpy array with an empty list in a different file. I will dig into the leak in a different place and vote to delete this question as I find it too specific to assist anyone else.

[Update 1]: New question describing the source of the problem:
Why does comparison of a numpy array with a list consume so much memory?

                        
这篇关于在长时间运行的应用程序中减少numpy内存占用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！