追踪内存泄漏？ [英] Tracking down memory leaks?
问题描述
我有一个名为compute的函数的应用程序，它给出了一个
文件名，遍历该文件并执行各种统计
分析。它广泛使用数组并循环很多。它打印出标准输出的统计显着性测试的结果
。由于
计算函数返回并且我认为没有使用全局范围的变量
，我认为当它发生时，所有内存都返回
回到操作系统。
相反，我看到的是每次迭代都使用了几个megs。
例如，python使用52 megs开始时，它经历了几次迭代
我突然使用超过500兆的内存。
有没有人有任何指示如何弄清楚我在做什么
错了？
谢谢，
mohan
Em Dom，2006-02-12？* s 05:11 -0800，MKoool escreveu：我有一个应用程序名为compute的函数，它给出了一个
文件名，遍历该文件并执行各种统计分析。它广泛使用数组并循环很多。它打印出标准输出的统计显着性测试结果。由于计算函数返回并且我认为没有使用全局范围的变量
，我认为当它执行时，所有内存都会返回给操作系统。
相反，我看到的是每次迭代都会使用几个megs。
例如，python在启动时使用52 megs，它经历了几次迭代，我突然使用了更多超过500兆的公羊。
有没有人有任何关于如何弄清楚我在做什么错误的指示？
您是否尝试强制进行垃圾回收？例如，每次函数返回时尝试运行
gc.collect（）。见
http：//www.python .org / doc / current / lib / module-gc.html 了解更多详情。
谢谢，
mohan
Cya，
Felipe。
-
Quem excele em empregar a for？§amilitarsubjulga os ex rcitos dos
outros povos sem travar batalha，toma cidades fortificadas dos outros
povos sem as atacar e destr？3i os estados dos outros povos sem lutas
prolongadas。 Deve lutar sob o C ?? u com o prop？3sito primordial da
''preserva？§？£o''。 Desse modo suas armas n？£se se embotar？£o，e os ganhos
poder？£o ser preservados。 Essa ?? a estrat ?? gia para planejar ofensivas。
- Sun Tzu，emA arte da guerra
我认为* Python使用引用计数进行垃圾收集。我已经听说过想要改变它的人（标记和扫描？）。
无论如何，Python为每个对象存储一个计数器。每当你对一个对象进行
引用时，这个计数器就会增加。每次指针删除或重新分配对象时，计数器都会减少。
当计数器达到零时，对象将从内存中释放。使用此算法的缺陷
如果您创建循环引用，则永远不会释放
对象。在删除头部
指针后，尾部指向
头的链表将在每个节点的引用计数为1。所以列表永远不会被释放。确保你没有
创建一个循环引用。这样的事情：
a = [1,2,3,4,5,6]
b = [''a''，''b' '，''c''，'d'']
c = [10,20,30,40]
a [3] = b
b [1] = c
c [0] = a
最后一次分配创建一个循环的refence，直到它<删除了
，这些对象中的任何一个都将从内存中删除。
我不是python内部的专家，而且他们可能会/>
有办法检查这样的情况。我认为深度镜检查
方法可以捕捉到这一点，但我不认为*基本的垃圾收集看起来像这样的东西是
。
David
On Sun，2006年2月12日05:11:02 -0800，MKoool写道：
我有一个名为compute的函数的应用程序，它给出了一个
文件名，遍历该文件并执行各种统计分析。它广泛使用数组并循环很多。它打印出标准输出的统计显着性测试结果。由于计算函数返回并且我认为没有使用全局范围的变量
，我认为当它执行时，所有内存都会返回到操作系统。
我可能会弄错，如果是的话我会欢迎更正，但是Python
在它终止之前不会将内存返回给操作系统。 />
对象在收集垃圾时将内存返回给Python，但不是操作系统的
.
相反，我是什么看，每次迭代都会使用几个megs。
例如，python在启动时使用52 megs，经过几次迭代，我突然使用超过500 meg的内存。 />
有没有人有任何关于如何弄清楚我在做什么
错误的指示？
你有多大的档案正在读书？如果它是（比方说）400 MB，那么它就是
，你将使用500MB的内存并不奇怪。如果文件是25K，
这是另一个故事。
如何在处理数据时存储数据？我会看到
隐藏的副本。
我建议你重新计算你的程序。而不是一个巨大的功能，打破它的大量小的，并从计算中调用它们。是的，这个
将使用更多的内存，当你试图使用更少的内存时，这可能听起来适得其反;但从长远来看它
将允许您的计算机更有效地使用内存（因为它们需要比一个巨大的功能更容易
页面小功能），并且它
将更容易编写和调试，当你可以在单个函数中隔离任务的各个部分时。
重新分解将还有另一个好处：你可能会自己找到问题
。
-
史蒂文。
I have an application with one function called "compute", which given a
filename, goes through that file and performs various statistical
analyses. It uses arrays extensively and loops alot. it prints the
results of it''s statistical significance tests to standard out. Since
the compute function returns and I think no variables of global scope
are being used, I would think that when it does, all memory returns
back to the operating system.
Instead, what I see is that every iteration uses several megs more.
For example, python uses 52 megs when starting out, it goes through
several iterations and I''m suddenly using more than 500 megs of ram.
Does anyone have any pointers on how to figure out what I''m doing
wrong?
Thanks,
mohan
Em Dom, 2006-02-12 ?*s 05:11 -0800, MKoool escreveu:I have an application with one function called "compute", which given a
filename, goes through that file and performs various statistical
analyses. It uses arrays extensively and loops alot. it prints the
results of it''s statistical significance tests to standard out. Since
the compute function returns and I think no variables of global scope
are being used, I would think that when it does, all memory returns
back to the operating system.
Instead, what I see is that every iteration uses several megs more.
For example, python uses 52 megs when starting out, it goes through
several iterations and I''m suddenly using more than 500 megs of ram.
Does anyone have any pointers on how to figure out what I''m doing
wrong?
Have you tried to force a garbage collection? Try, for example, running
gc.collect() everytime the function returns. See
http://www.python.org/doc/current/lib/module-gc.html for more details.
Thanks,
mohan
Cya,
Felipe.
--
"Quem excele em empregar a for?§a militar subjulga os ex??rcitos dos
outros povos sem travar batalha, toma cidades fortificadas dos outros
povos sem as atacar e destr?3i os estados dos outros povos sem lutas
prolongadas. Deve lutar sob o C??u com o prop?3sito primordial da
''preserva?§?￡o''. Desse modo suas armas n?￡o se embotar?￡o, e os ganhos
poder?￡o ser preservados. Essa ?? a estrat??gia para planejar ofensivas."
-- Sun Tzu, em "A arte da guerra"
I *think* Python uses reference counting for garbage collection. I''ve
heard talk of people wanting to change this (to mark and sweep?).
Anyway, Python stores a counter with each object. Everytime you make a
reference to an object this counter is increased. Everytime a pointer
to the object is deleteted or reassigned the counter is decreased.
When the counter reaches zero the object is freed from memory. A flaw
with this algorithm is that if you create a circular reference the
object will never be freed. A linked list where the tail points to the
head will have a reference count of 1 for each node, after the head
pointer is deleted. So the list is never freed. Make sure you are not
creating a circular reference. Something like this:
a = [1, 2, 3, 4, 5, 6]
b = [''a'', ''b'', ''c'', ''d'']
c = [10, 20, 30, 40]
a[3] = b
b[1] = c
c[0] = a
the last assignment creates a circular refence, and until it is
removed, non of these objects will be removed from memory.
I''m not an expert on python internals, and it is possible that they
have a way of checking for cases like this. I think the deepcopy
method catches this, but I don''t *think* basic garbage collection look
for this sort of thing.
David
On Sun, 12 Feb 2006 05:11:02 -0800, MKoool wrote:
I have an application with one function called "compute", which given a
filename, goes through that file and performs various statistical
analyses. It uses arrays extensively and loops alot. it prints the
results of it''s statistical significance tests to standard out. Since
the compute function returns and I think no variables of global scope
are being used, I would think that when it does, all memory returns
back to the operating system.
I may be mistaken, and if so I will welcome the correction, but Python
does not return memory to the operating system until it terminates.
Objects return memory to Python when they are garbage collected, but not
the OS.
Instead, what I see is that every iteration uses several megs more.
For example, python uses 52 megs when starting out, it goes through
several iterations and I''m suddenly using more than 500 megs of ram.
Does anyone have any pointers on how to figure out what I''m doing
wrong?
How big is the file you are reading in? If it is (say) 400 MB, then it is
hardly surprising that you will be using 500MB of RAM. If the file is 25K,
that''s another story.
How are you storing your data while you are processing it? I''d be looking
for hidden duplicates.
I suggest you re-factor your program. Instead of one giant function, break
it up into lots of smaller ones, and call them from compute. Yes, this
will use a little more memory, which might sound counter-productive at the
moment when you are trying to use less memory, but in the long term it
will allow your computer to use memory more efficiently (it is easier to
page small functions as they are needed than one giant function), and it
will be much easier for you to write and debug when you can isolate
individual pieces of the task in individual functions.
Re-factoring will have another advantage: you might just find the problem
on your own.
--
Steven.
这篇关于追踪内存泄漏？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
