发现此进程当前内存使用情况的最佳方式，跨平台？ [英] best way to discover this process's current memory usage, cross-platform?

问题描述

在C编码中修复了内存泄漏（不是Python引用的漏洞，某些其他东西，我在某些情况下没有正常释放）

扩展我维护，我需要一种方法来测试泄漏确实是固定的。匆忙，我最初使用的是q& d hack ...：

if sys.platform in（''linux2''，''darwin''）：

def _memsize（）：

"""这个函数试图返回一个测量这个过程消耗多少内存的时间，在某个任意单位（如果它不是b
，它会返回0） 。

""

gc.collect（）

试试：

x = int（os .popen（''ps -p％d -o vsz | tail -1''％os.getpid（））。read（））

除了：

x = 0

返回x

否则：

def _memsize（）：

返回0


有一个_memsize（）函数可用，测试然后执行：

before = _memsize（）

＃很多重复执行的代码应该不消费

＃任何净内存，但习惯在泄漏时那里

之后= _memsize（）

并检查后==之前。


然而，_memsize只是一个太大的黑客，我真的想要清空它。它也不够跨平台。此外，我在一个Linux平台上的用户收到了一个错误

的报告，这个报告与我测试过的用户不同，而且它归结为偶尔这样的事实。在他的

机器上它转过来之后是+4之前（对于上述评论中任何大量的

重复的代码） - 我不是确定

度量单位应该是多少（可能是512字节的块，页面是

大小为2048？无论如何......），但显然是额外的页面正在使用

某个地方。


所以，我以为我会转向人群的智慧......你们怎么样？

开始添加你的自动回归测试，检查某些内存泄漏没有再次发生，因为跨平台可行吗？

特别是，你如何编码_memsize（）跨平台？ （我可以

如果需要可以轻松使用C而不是Python，将其作为辅助

函数添加到现有扩展中进行测试）。

TIA，


Alex

解决方案

不确定我是否应该开始一个新线程，但是

因为这是密切相关的，我会保持原样。


Alex Martelli写道：

在我维护的C编码扩展中修复了内存泄漏（不是Python引用的泄漏，某些其他东西，我在某些情况下没有正常释放），我需要一个测试泄漏确实已经解决的方法。

我想研究一下内存使用了多少内存

Python对象。我的动机是98％的纯智力

好​​奇心和2％的优化。


我想知道我是否可以这样做：

obj = something（）

bytes_used = sizeof（obj）


（显然没有内置函数sizeof ...

等等，让我检查......不，不是内置的）


我已经阅读了gc和pdb的文档而且没有任何内容

给我做这样的事情。

-

史蒂文。

Steven D''Aprano< st *** @ REMOVEMEcyber.com.au>写道：

不确定我是否应该开始一个新线程，但
因为这是密切相关的，我会保持原样。
Alex Martelli写道：

修复了内存泄漏（不是Python参考的泄漏，一些其他东西我没有正确释放我维护的C编码扩展中的某些情况）我需要一种方法来测试泄漏确实已经修复。

我想调查一下
Python对象使用了多少内存。我的动机是98％纯粹的知识分子好奇心和2％的优化。

我相信这是PySizer项目的目的（其中一个） ; Google

Summer of Code项目，最近在这个小组宣布了

（我确定任何搜索引擎都能指引你，无论如何）。


我没有检查过，因为我的目的不同 - 我的是

根本不是与Python相关的泄漏，只是一个在C代码中泄漏（其中
恰巧是Python扩展模块）。

Alex

Alex Martelli写道：

所以，我以为我会转向人群的智慧......你们怎么会加入你的自动化回归测试一个检查某个内存泄漏没有再次发生的问题，就像跨平台一样可行吗？
特别是，你如何编写_memsize（）"横platformly" ;? （如果需要，我可以轻松使用C而不是Python，将其作为辅助功能添加到现有扩展中进行测试）。

如果您正在使用Unix，可以使用getrusage（2）吗？

import resource
r = resource.getrusage（resource.RUSAGE_SELF）
print r [2：5]

我的gentoo上有零amd64盒子。不知道为什么。我想也许它是b
是Python，但是C给出了相同的结果。


另一种可能性是调用sbrk（0），它应该返回

堆。然后，您可以返回此值并进行检查。它需要一个很小的C模块，但它应该很容易并适用于大多数unix。您可以通过将它与id（0）进行比较来确定方向堆的增长，这应该在解释器生命的早期就已经分配了

。


我意识到这并不完美，因为内存变得支离破碎，但可能会工作。从2.3及以后使用pymalloc，碎片可能不是很多问题。由于内存分配在一个大块中，然后必须发布为

。


这些技术可能适用于Windows，但有一些注意事项。如果您对Windows感兴趣，请参阅：
http://msdn.microsoft.com/library/de...l/UCMGch09.asp


但是不能想到任何万无一失的事。


HTH，

n

Having fixed a memory leak (not the leak of a Python reference, some
other stuff I wasn''t properly freeing in certain cases) in a C-coded
extension I maintain, I need a way to test that the leak is indeed
fixed. Being in a hurry, I originally used a q&d hack...:
if sys.platform in (''linux2'', ''darwin''):
def _memsize():
""" this function tries to return a measurement of how much memory
this process is consuming, in some arbitrary unit (if it doesn''t
manage to, it returns 0).
"""
gc.collect()
try:
x = int(os.popen(''ps -p %d -o vsz|tail -1'' % os.getpid()).read())
except:
x = 0
return x
else:
def _memsize():
return 0

Having a _memsize() function available, the test then does:
before = _memsize()
# a lot of repeated executions of code that should not consume
# any net memory, but used to when the leak was there
after = _memsize()
and checks that after==before.

However, that _memsize is just too much of a hack, and I really want to
clean it up. It''s also not cross-platform enough. Besides, I got a bug
report from a user on a Linux platform different from those I had tested
myself, and it boils down to the fact that once in a while on his
machine it turns our that after is before+4 (for any large number of
repetitions of the code in the above comment) -- I''m not sure what the
unit of measure is supposed to be (maybe blocks of 512 byte, with a page
size of 2048? whatever...), but clearly an extra page is getting used
somewhere.

So, I thought I''d turn to the "wisdom of crowds"... how would YOU guys
go about adding to your automated regression tests one that checks that
a certain memory leak has not recurred, as cross-platform as feasible?
In particular, how would you code _memsize() "cross-platformly"? (I can
easily use C rather than Python if needed, adding it as an auxiliary
function for testing purposes to my existing extension).
TIA,

Alex

解决方案

Not sure if I should start a new thread or not, but
since this is closely related, I''ll just leave it as is.

Alex Martelli wrote:

Having fixed a memory leak (not the leak of a Python reference, some
other stuff I wasn''t properly freeing in certain cases) in a C-coded
extension I maintain, I need a way to test that the leak is indeed
fixed.

I would like to investigate how much memory is used by
Python objects. My motive is 98% pure intellectual
curiosity and 2% optimization.

I wonder whether I can do something like this:

obj = something()
bytes_used = sizeof(obj)

(obviously there is no built-in function sizeof...
wait, let me check... nope, not a built-in)

I''ve read the docs for gc and pdb and nothing stands
out to me as doing anything like this.
--
Steven.

Steven D''Aprano <st***@REMOVEMEcyber.com.au> wrote:

Not sure if I should start a new thread or not, but
since this is closely related, I''ll just leave it as is.

Alex Martelli wrote:

Having fixed a memory leak (not the leak of a Python reference, some
other stuff I wasn''t properly freeing in certain cases) in a C-coded
extension I maintain, I need a way to test that the leak is indeed
fixed.

I would like to investigate how much memory is used by
Python objects. My motive is 98% pure intellectual
curiosity and 2% optimization.

I believe that''s the purpose of the PySizer project (one of the "Google
Summer of Code" projects), which was recently announced on this group
(I''m sure any search engine will be able to direct you to it, anyway).

I have not checked it out, because my purpose is different -- mine is
not a Python-related leak at all, just a leak within C code (which
happens coincidentally to be a Python extension module).
Alex

Alex Martelli wrote:

So, I thought I''d turn to the "wisdom of crowds"... how would YOU guys
go about adding to your automated regression tests one that checks that
a certain memory leak has not recurred, as cross-platform as feasible?
In particular, how would you code _memsize() "cross-platformly"? (I can
easily use C rather than Python if needed, adding it as an auxiliary
function for testing purposes to my existing extension).

If you are doing Unix, can you use getrusage(2)?

import resource
r = resource.getrusage(resource.RUSAGE_SELF)
print r[2:5]

I get zeroes on my gentoo amd64 box. Not sure why. I thought maybe it
was Python, but C gives the same results.

Another possibiity is to call sbrk(0) which should return the top of
the heap. You could then return this value and check it. It requires
a tiny C module, but should be easy and work on most unixes. You can
determine direction heap grows by comparing it with id(0) which should
have been allocated early in the interpreters life.

I realize this isn''t perfect as memory becomes fragmented, but might
work. Since 2.3 and beyond use pymalloc, fragmentation may not be much
of an issue. As memory is allocated in a big hunk, then doled out as
necessary.

These techniques could apply to Windows with some caveats. If you are
interested in Windows, see:
http://msdn.microsoft.com/library/de...l/UCMGch09.asp

Can''t think of anything fool-proof though.

HTH,
n

这篇关于发现此进程当前内存使用情况的最佳方式，跨平台？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！