Python中的内存泄漏Twisted：它在哪里？ [英] Memory leak in Python Twisted: where is it?

问题描述

我有一个Twisted服务器正在加载。当服务器处于负载下时，内存使用率增加，并且永远不会回收（即使没有更多客户端）。下一次进入高负载时，内存使用量再次增加。以下是当时情况的快照： b
$ b

• RSS内存为400 MB（通常最大客户端数应为200 MB）。


• gc.garbage 为空，因此没有无法收集的对象。 使用 objgraph.py
strong>显示没有明显的漏洞候选人（正常，健康流程和漏洞流程之间没有显着差异）。 使用 pympler 显示几十MB （仅限于）Python对象（主要是词典，列表，str和其他本机容器）使用。


• Valgrind with leak-check = full enabled doesn没有显示任何主要泄漏（只有几个MB'明确丢失'） - 所以C扩展不是罪魁祸首。总内存也不会加上顶部显示的400MB + ：
  
  

  == 23072 == HEAP摘要：

== 23072 ==在退出时使用：65650760个字节在463,153个块中

== 23072 ==总堆使用量：124,269,475个分配，123,806,322个释放，32,660,215,602个字节分配

我能找到的唯一解释是某些对象没有被垃圾收集器，因此它们不会被objgraph和pympler显示出来，而是使用大量的RAM。

我还有其他什么工具或解决方案？编译Python解释器时，通过使用sys.getobjects来帮助编译Python解释器？解析方案

如果代码只是在负载下泄漏你是否证实了这一点？），我会看看消息缓冲的所有位置。流程本身的内存使用量是否增加？或者系统的内存使用量增加了？如果是后一种情况，你的服务器可能太慢而无法跟上传入的消息并且操作系统缓冲区已满。

I have a Twisted server under load. When the server is under load, memory usage increases, and it is never reclaimed (even when there are no more clients). Next time it goes into high load, memory usage increases again. Here's a snapshot of the situation at that point:

• RSS memory is 400 MB (should be 200MB with usual max number of clients).
• gc.garbage is empty, so there are no uncollectable objects.
• Using objgraph.py shows no obvious candidates for leaks (no notable difference between a normal, healthy process and a leaking process).
• Using pympler shows a few tens of MB (only) used by Python objects (mostly dict, list, str and other native containers).

• Valgrind with leak-check=full enabled doesn't show any major leaks (only couple of MBs 'definitively lost') - so C extensions are not the culprit. The total memory also doesn't add up with the 400MB+ shown by top:

  ==23072== HEAP SUMMARY:
==23072== in use at exit: 65,650,760 bytes in 463,153 blocks
==23072== total heap usage: 124,269,475 allocs, 123,806,322 frees, 32,660,215,602 bytes allocated

The only explanation I can find is that some objects are not tracked by the garbage collector, so that they are not shown by objgraph and pympler, yet use an enormous amount of RAM.

What other tools or solutions do I have? Would compiling the Python interpreter in debug mode help, by using sys.getobjects?

解决方案

If the code is only leaking under load (did you verify this?), I'd have a look at all spots where messages are buffered. Does the memory usage of the process itself increase? Or does the memory use of the system increase? If it's the latter case, your server might simply be too slow to keep up with the incoming messages and the OS buffer fill up..

这篇关于Python中的内存泄漏Twisted：它在哪里？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！