狩猎内存泄漏 [英] Hunting a memory leak

问题描述

我正试图在我的程序中发现内存泄漏。我已经采取了几种方法，但泄漏仍然无法出现。


首先，我试图使用垃圾收集器寻找

无法收集的物品。我已经使用了下一个：


＃代码开头

gc.enable（）

gc.set_debug （gc.DEBUG_LEAK）


< snip>


＃代码末尾：

print" ; \\\
GARBAGE："

gc.collect（）


print" \\\
GARBAGE OBJECTS："

对于gc.garbage中的x：

s = str（x）

打印类型（x），" \ n"，s

有了这个，我没有垃圾对象。


然后我采取了我在python开发人员名单中看到过的方法

由Walter D？rwald提供，基本上是创建一个

调试版本的python，用泄漏的代码创建一个unitest，并且

修改unittest.py提取在该代码中计算的总参考值增量

（参见
http：//aspn.activestat e.com/ASPN/Mai...n-dev/1770868）。


有了这个，我看到我的引用计数每次增加一/>
测试执行。但问题是：是否有某种方法可以查看泄漏的

对象（或进行内存转储）？


我已经使用valgrind（ http://developer.kde.org/~sewardj/ ）看看它是否可以检测到泄漏。事实上，它检测到了一堆，但我害怕他们与我正在寻找的泄漏无关。我说是因为当我循环泄漏我的漏码时，valgrind总是报告相同数量的泄漏内存，与
$的数量无关。 b $ b迭代（顶部告诉我内存使用正在增长！）。


我的代码在C中使用扩展模块，所以我恐怕不会这样。 b $ b有助于缓解这个问题。我认为所有的malloc都是正确释放的，但是我不能确定（但是，valgrind没有

在扩展中没有检测到任何错误）。


对不起，但是我对这段代码更加明确，因为它是非常复杂的（它是PyTables软件包， http://pytables.sf.net），

我无法做到简单待发布的例子

。但是，如果有人想看看代码，你可以从

下载它（ http://sourceforge.net/project/showf...roup_id=63486) 。我是

附上一个暴露泄漏的单位测试。


我有点绝望。任何提示？


Francesc Alted


-


＃Unittest暴露内存泄漏

导入系统

导入unittest

导入os

导入tempfile

$ b来自表格导入的$ b *

＃下一次导入只需要这个测试套件

#from表导入组，叶，表，数组

verbose = 0

类WideTreeTestCase（unittest.TestCase）：


def test00_Leafs（个体经营）：


导入时间

maxchilds = 2

如果详细：

打印''\ n''，' ' - =''* 30

print" Running％s.test00_wideTree ..." ％\

self .__ class __.__ name__

print"测试的最大孩子数量："，maxchilds

＃打开一个新的空HDF5文件

file = tempfile.mktemp（" .h5"）

#file =" test_widetree.h5"


fileh = openFile（file，mode =" w"）

如果详细：

print"孩子写进展："，

范围内的孩子（maxchilds）：

如果详细：

print"％3d，" ％（孩子），

a = [1,1]

fileh.createGroup（fileh.root，''group''+ str（child），

" child：％d"％child）

＃注释createArray调用以查看泄漏消息

fileh.createArray（" / group" + str（child），''array''+ str（child），

a，" child：％d"％child）

if verbose：

打印

＃关闭文件

fileh.close（）

＃----------- -------------------------------------------------- ---------


def suite（）：

theSuite = unittest.TestSuite（）

theSuite .addTest（unittest.makeSuite（WideTreeTestCa se））


返回theSuite

if __name__ ==''__ main__''：

unittest.main（defaultTest =''suite''）

Hi,

I''m trying to discover a memory leak on a program of mine. I''ve taken
several approaches, but the leak still resists to appear.

First of all, I''ve tried to use the garbage collector to look for
uncollectable objects. I''ve used the next:

# at the beginning of code
gc.enable()
gc.set_debug(gc.DEBUG_LEAK)

<snip>

# at the end of code:
print "\nGARBAGE:"
gc.collect()

print "\nGARBAGE OBJECTS:"
for x in gc.garbage:
s = str(x)
print type(x),"\n ", s

With that, I get no garbage objects.

Then I''ve taken an approach that I''ve seen in python developers list
contributed by Walter D?rwald, that basically consists in creating a
debug version of python, create a unitest with the leaking code, and
modify the unittest.py to extract the increment of total reference
counting in that code (see
http://aspn.activestate.com/ASPN/Mai...n-dev/1770868).

With that, I see that my reference count grows by one each time the
test execute. But the problem is: is there some way to look at the
object (or make a memory dump) that is leaking?.

I''ve used valgrind (http://developer.kde.org/~sewardj/) to see if it
could detect the leak. In fact, it detects a bunch of them, but I am
afraid that they are not related with the leak I''m looking for. I am
saying that because, when I loop over my leaky code, valgrind always
report the same amount of leaky memory, independently of the number of
iterations (while top is telling me that memory use is growing!).

My code uses extension modules in C, so I am afraid this does not
contribute to alleviate the problem. I think all the malloc are
correctly freed, but I can''t be sure (however, valgrind does not
detect nothing wrong in the extension).

I am sorry, but I cannot be more explicit about the code because it
is quite complex (it is the PyTables package, http://pytables.sf.net),
and I was unable to make a simple example to be published
here. However, if anyone is tempted to have a look at the code, you
can download it from
(http://sourceforge.net/project/showf...roup_id=63486). I am
attaching a unittest that exposes the leak.

I am a bit desperate. Any hint?

Francesc Alted

--

# Unittest to expose the memory leak
import sys
import unittest
import os
import tempfile

from tables import *
# Next imports are only necessary for this test suite
#from tables import Group, Leaf, Table, Array

verbose = 0

class WideTreeTestCase(unittest.TestCase):

def test00_Leafs(self):

import time
maxchilds = 2
if verbose:
print ''\n'', ''-='' * 30
print "Running %s.test00_wideTree..." % \
self.__class__.__name__
print "Maximum number of childs tested :", maxchilds
# Open a new empty HDF5 file
file = tempfile.mktemp(".h5")
#file = "test_widetree.h5"

fileh = openFile(file, mode = "w")
if verbose:
print "Children writing progress: ",
for child in range(maxchilds):
if verbose:
print "%3d," % (child),
a = [1, 1]
fileh.createGroup(fileh.root, ''group'' + str(child),
"child: %d" % child)
# Comment the createArray call to see the leak disapear
fileh.createArray("/group" + str(child), ''array'' + str(child),
a, "child: %d" % child)
if verbose:
print
# Close the file
fileh.close()
#----------------------------------------------------------------------

def suite():
theSuite = unittest.TestSuite()
theSuite.addTest(unittest.makeSuite(WideTreeTestCa se))

return theSuite
if __name__ == ''__main__'':
unittest.main(defaultTest=''suite'')

推荐答案

Debian用户< fa **** @ inspiron。 openlc.org>写道：

Debian User <fa****@inspiron.openlc.org> writes:

我正试图在我的程序中发现内存泄漏。我已经采取了几种方法，但泄漏仍然无法出现。

首先，我试图使用垃圾收集器寻找
无法收集对象。


[snip]

然后我采取了我在python开发人员名单中看到过的方法
由Walter D提供的？ rwald，基本上包括创建一个python的调试版本，用泄漏的代码创建一个unitest，并修改unittest.py以提取该代码中的总引用的增量
（参见
http：// aspn .activestate.com / ASPN /麦... N-dev /目录1770868）。


好​​吧，同一个帖子中的某个地方是对

TrackRefs类的各种引用。你尝试过使用它吗？它应该告诉你什么

类型的对象泄漏，这是一个好的开始。

有了这个，我看到我的引用计数每次增长一次/>测试执行。但问题是：是否有某种方法可以查看泄漏的
对象（或进行内存转储）？


见上文:-)

我用过valgrind（ http://developer.kde.org/~sewardj/ ）看看它是否可以检测到泄漏。事实上，它检测到了一堆，但我担心它们与我正在寻找的泄漏无关。我说是因为，当我循环泄漏我的泄密代码时，valgrind总是报告相同数量的泄漏内存，与
迭代次数无关（顶部告诉我内存）使用正在增长！）。


在Python程序结束时，有各种各样的东西（实习字符串，f''ex）总是倾向于生存：这些只是在一个非常扭曲的感觉中泄漏了。


我不知道是否有办法让vaglrind告诉你什么是<分配但是没有在两个任意程序点之间解除分配的


执行。

我的代码在C中使用扩展模块，所以我担心这不会
有助于缓解这个问题。


好​​吧，在所有可能的情况下，该错误都在C扩展模块中。你试过在调试器中单步调试代码吗？有时'那是'

a发现逻辑错误的好方法。

对不起，但我不能更清楚地看到代码，因为它
非常复杂（它是PyTables包， http://pytables.sf.net），
我无法在这里发表一个简单的例子。
但是，如果有人想看看代码，你可以从
下载它（ http://sourceforge.net/project/showf...roup_id=63486) 。我附上了暴露泄漏的单位测试。

我有点绝望。任何提示？

I''m trying to discover a memory leak on a program of mine. I''ve taken
several approaches, but the leak still resists to appear.

First of all, I''ve tried to use the garbage collector to look for
uncollectable objects.
[snip]
Then I''ve taken an approach that I''ve seen in python developers list
contributed by Walter D?rwald, that basically consists in creating a
debug version of python, create a unitest with the leaking code, and
modify the unittest.py to extract the increment of total reference
counting in that code (see
http://aspn.activestate.com/ASPN/Mai...n-dev/1770868).
Well, somewhere in that same thread are various references to a
TrackRefs class. Have you tried using that? It should tell you what
type of object is leaking, which is a good start.
With that, I see that my reference count grows by one each time the
test execute. But the problem is: is there some way to look at the
object (or make a memory dump) that is leaking?.
See above :-)
I''ve used valgrind (http://developer.kde.org/~sewardj/) to see if it
could detect the leak. In fact, it detects a bunch of them, but I am
afraid that they are not related with the leak I''m looking for. I am
saying that because, when I loop over my leaky code, valgrind always
report the same amount of leaky memory, independently of the number of
iterations (while top is telling me that memory use is growing!).
There are various things (interned strings, f''ex) that always tend to
be alive at the end of a Python program: these are only leaks in a
very warped sense.

I don''t know if there''s a way to get vaglrind to tell you what''s
allocated but not deallocated between two arbitrary points of program
execution.
My code uses extension modules in C, so I am afraid this does not
contribute to alleviate the problem.
Well, in all likelyhood, the bug is IN the C extension module. Have
you tried stepping through the code in a debugger? Sometime''s that''s
a good way of spotting a logic error.
I am sorry, but I cannot be more explicit about the code because it
is quite complex (it is the PyTables package, http://pytables.sf.net),
and I was unable to make a simple example to be published
here. However, if anyone is tempted to have a look at the code, you
can download it from
(http://sourceforge.net/project/showf...roup_id=63486). I am
attaching a unittest that exposes the leak.

I am a bit desperate. Any hint?

不是。尝试使用TrackRef。


干杯，

mwh


-

我我将要搜索谷歌的合同刺客去Iomega

和惠普的编程小组并用某种方式杀死那里的每个人

带电的生锈带刺的东西。

- http://bofhcam.org/journal /journal.html ，2002-01-08

Not really. Try using TrackRefs.

Cheers,
mwh

--
I''m about to search Google for contract assassins to go to Iomega
and HP''s programming groups and kill everyone there with some kind
of electrically charged rusty barbed thing.
-- http://bofhcam.org/journal/journal.html, 2002-01-08

>我正试图在我的程序中发现内存泄漏。我已经采取了

> I''m trying to discover a memory leak on a program of mine. I''ve taken

几种方法，但泄漏仍然无法出现。

several approaches, but the leak still resists to appear.

首先，单步执行C代码令人惊讶有效。我衷心地说

推荐它。


如果您真的绝望，可以使用以下一些想法。您将需要

来完成一些工作，以使它们在您的情况下有用。


1.跟踪所有新创建的对象。警告：

中使用的id技巧此代码不正确，因为新分配的对象可以与旧对象具有相同的

地址，因此您应该通过创建更好的方法来设计更多

唯一哈希值。或者只是按原样使用代码，看看是否无效。代码

告诉你一些事情;-)

全局lastObjectsDict

objects = gc.get_objects（）


newObjects = [o表示对象中的o如果不是lastObjectsDict.has_key（id（o））]


lastObjectsDict = {}

对象中的o：

lastObjectsDict [id（o）] = o


2.跟踪对象的数量。


def printGc（message = None，onlyPrintChanges = false）：


如果不是debugGC：返回无


如果不是消息：

message = callerName（n = 2）#Le left as a exercise for reader。


global lastObjectCount


尝试：

n = len（gc.garbage）

n2 = len（gc.get_objects（））

delta = n2- lastObjectCount

如果不只是打印更改或增量：

如果n：

print" garbage：％d，objects：％+ 6d =％7d ％S" ％

（n，delta，n2，message）

else：

print" objects：％+ 6d =％7d％s" ％

（n2-lastObjectCount，n2，消息）


lastObjectCount = n2

返回delta

除外：

traceback.print_exc（）

返回无


3.打印大量信息...


def printGcRefs（verbose = true）：


refs = gc.get_referrers（app（）。windowList [0]）

print'' - ''* 30


如果详细：

print" refs of，app（）。windowList [0]

参考参考：

打印类型（参考）

如果0：＃very verbose

if type （ref）== type（{}）：

keys = ref.keys（）

keys.sort（）

for key in键：

val = ref [key]

if isinstance（val，leoFrame.LeoFrame）：＃更改为

需要

打印键，参考号[key]

else：

print"％d referers" ％len（refs）


这里app（）。windowList是我的应用程序的关键数据结构。将你的

替换为新的参数。


基本上，Python会为你提供所需的所有信息。问题

是有太多的信息，所以你必须尝试过滤

它。不要惊慌：你可以做到。


4.一种完全不同的方法。考虑这个功能：


def clearAllIvars（o）：


"""清除o的所有ivars，某些成员等级。"""


o .__ dict __。clear（）


此功能将混凝土墙磨成砂粒。 GC将

然后单独收回每个谷物。


我的应用程序包含几个相互引用的类。而不是

跟踪所有联锁参考，当需要删除

主数据结构时，我的应用程序只需调用clearAllIvars以获取各种

类。当然，需要一些注意，以确保拨打正确的订单

。


HTH。


爱德华

---------------------------------------- ----------------------------

Edward K. Ream电子邮件： ed ******* @ charter.net

Leo：包含轮廓的文学编辑器

Leo： http://webpages.charter.net/edreamleo/front.html

-------------------------------------- ------------------------------

First, single-stepping through C code is surprisingly effective. I heartily
recommend it.

Here are some ideas you might use if you are truly desperate. You will have
to do some work to make them useful in your situation.

1. Keep track of all newly-created objects. Warning: the id trick used in
this code is not proper because newly allocated objects can have the same
address as old objects, so you should devise a better way by creating a more
unique hash. Or just use the code as is and see whether the "invalid" code
tells you something ;-)

global lastObjectsDict
objects = gc.get_objects()

newObjects = [o for o in objects if not lastObjectsDict.has_key(id(o))]

lastObjectsDict = {}
for o in objects:
lastObjectsDict[id(o)]=o

2. Keep track of the number of objects.

def printGc(message=None,onlyPrintChanges=false):

if not debugGC: return None

if not message:
message = callerName(n=2) # Left as an exercise for the reader.

global lastObjectCount

try:
n = len(gc.garbage)
n2 = len(gc.get_objects())
delta = n2-lastObjectCount
if not onlyPrintChanges or delta:
if n:
print "garbage: %d, objects: %+6d =%7d %s" %
(n,delta,n2,message)
else:
print "objects: %+6d =%7d %s" %
(n2-lastObjectCount,n2,message)

lastObjectCount = n2
return delta
except:
traceback.print_exc()
return None

3. Print lots and lots of info...

def printGcRefs (verbose=true):

refs = gc.get_referrers(app().windowList[0])
print ''-'' * 30

if verbose:
print "refs of", app().windowList[0]
for ref in refs:
print type(ref)
if 0: # very verbose
if type(ref) == type({}):
keys = ref.keys()
keys.sort()
for key in keys:
val = ref[key]
if isinstance(val,leoFrame.LeoFrame): # changes as
needed
print key,ref[key]
else:
print "%d referers" % len(refs)

Here app().windowList is a key data structure of my app. Substitute your
own as a new argument.

Basically, Python will give you all the information you need. The problem
is that there is way too much info, so you must experiment with filtering
it. Don''t panic: you can do it.

4. A totally different approach. Consider this function:

def clearAllIvars (o):

"""Clear all ivars of o, a member of some class."""

o.__dict__.clear()

This function will grind concrete walls into grains of sand. The GC will
then recover each grain separately.

My app contains several classes that refer to each other. Rather than
tracking all the interlocking references, when it comes time to delete the
main data structure my app simply calls clearAllIvars for the various
classes. Naturally, some care is needed to ensure that calls are made in
the proper order.

HTH.

Edward
--------------------------------------------------------------------
Edward K. Ream email: ed*******@charter.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------

2003-08- 29，Michael Hudson< mw*@python.net>写道：

On 2003-08-29, Michael Hudson <mw*@python.net> wrote:

Debian用户< fa **** @ inspiron.openlc.org>写道：

Debian User <fa****@inspiron.openlc.org> writes:

我正试图在我的程序中发现内存泄漏。我已经采取了几种方法，但泄漏仍然无法出现。

首先，我试图使用垃圾收集器寻找
无法收集对象。

I''m trying to discover a memory leak on a program of mine. I''ve taken
several approaches, but the leak still resists to appear.

First of all, I''ve tried to use the garbage collector to look for
uncollectable objects.

[snip]

[snip]

然后我采取了我在python开发人员列表中看到过的方法
由Walter D？rwald提供，主要是创建一个python的调试版本，用泄漏的代码创建一个unitest，并修改unittest.py以提取总引用的增量 http://aspn.activestate.com/ASPN/Mai...n-dev/1770868) 。

Then I''ve taken an approach that I''ve seen in python developers list
contributed by Walter D?rwald, that basically consists in creating a
debug version of python, create a unitest with the leaking code, and
modify the unittest.py to extract the increment of total reference
counting in that code (see
http://aspn.activestate.com/ASPN/Mai...n-dev/1770868).

嗯，同一个帖子中的某个地方是各种参考到一个TrackRefs类。你尝试过使用它吗？它应该告诉你什么类型的对象泄漏，这是一个好的开始。

Well, somewhere in that same thread are various references to a
TrackRefs class. Have you tried using that? It should tell you what
type of object is leaking, which is a good start.

有了这个，我看到我的引用计数每次增加一个/>测试执行。但问题是：是否有某种方法来查看泄漏的
对象（或进行内存转储）？。

With that, I see that my reference count grows by one each time the
test execute. But the problem is: is there some way to look at the
object (or make a memory dump) that is leaking?.

见上文： - ）

See above :-)

我使用了valgrind（ http：// developer.kde.org/~sewardj/ ）看看它是否可以检测到泄漏。事实上，它检测到了一堆，但我担心它们与我正在寻找的泄漏无关。我说是因为，当我循环泄漏我的泄密代码时，valgrind总是报告相同数量的泄漏内存，与
迭代次数无关（顶部告诉我内存）使用正在增长！）。

I''ve used valgrind (http://developer.kde.org/~sewardj/) to see if it
could detect the leak. In fact, it detects a bunch of them, but I am
afraid that they are not related with the leak I''m looking for. I am
saying that because, when I loop over my leaky code, valgrind always
report the same amount of leaky memory, independently of the number of
iterations (while top is telling me that memory use is growing!).

在Python程序结束时总是倾向于活着的各种东西（实习字符串，f''ex）：这些只是在非常扭曲的意义上泄漏。

我不知道是否有办法让迷失者告诉你分配了什么但没有在程序执行的两个任意点之间取消分配。

There are various things (interned strings, f''ex) that always tend to
be alive at the end of a Python program: these are only leaks in a
very warped sense.

I don''t know if there''s a way to get vaglrind to tell you what''s
allocated but not deallocated between two arbitrary points of program
execution.

我的代码在C中使用扩展模块，所以我担心这不会有助于缓解问题。

My code uses extension modules in C, so I am afraid this does not
contribute to alleviate the problem.

好吧，在所有可能的情况下，错误都在C扩展模块中。您是否尝试在调试器中单步执行代码？有时候'那是'找出逻辑错误的好方法。

Well, in all likelyhood, the bug is IN the C extension module. Have
you tried stepping through the code in a debugger? Sometime''s that''s
a good way of spotting a logic error.

对不起，但我不能更清楚地了解代码，因为它是
非常复杂（它是PyTables软件包， http://pytables.sf.net)，
我无法在这里发表一个简单的例子。
但是，如果有人想看看代码，你可以从
下载它（ http://sourceforge.net/project/showf...roup_id=63486) 。我附上了暴露泄漏的单位测试。

我有点绝望。任何提示？

I am sorry, but I cannot be more explicit about the code because it
is quite complex (it is the PyTables package, http://pytables.sf.net),
and I was unable to make a simple example to be published
here. However, if anyone is tempted to have a look at the code, you
can download it from
(http://sourceforge.net/project/showf...roup_id=63486). I am
attaching a unittest that exposes the leak.

I am a bit desperate. Any hint?

不是真的。尝试使用TrackRef。

干杯，
mwh

Not really. Try using TrackRefs.

Cheers,
mwh

这篇关于狩猎内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！