Python内存使用量没有下降 [英] Python Memory usage doesnt drop

问题描述

我正在编写一个扭曲的PB应用程序，该应用程序似乎使用了大量的内存，当用户断开连接时似乎永远不会释放.

我有一个客户端连接到的pb.Root对象，并调用一个返回pb.Referenceable对象的远程方法，该对象在创建以加快速度时会将大量信息读取到内存中(大约2GB的数据)行动.该对象以及有关客户端的其他一些信息将插入到列表中.

当客户端与服务器断开连接时，我将对此对象执行一些清理操作，以删除对正在存储的缓存对象的引用. chunkCache是​​我将数据存储在其中的命令.

def disconnected(self):
    self.connected = False
    self.chunkCache = None
    self.cur.close()

一旦客户端根据top断开了对内存的使用，它永远不会丢失2Gb.

我应该为此担心，还是在需要时释放分配的内存，或者是否有任何想法可以释放此内存?它是在创建对象时创建的，没有在其他任何地方传递.

在该对象中，我确实有一个deferToThread调用，这会阻止该项目被释放吗?

我正在Linux上使用python 2.7.

更新:

我很困惑，我刚刚向对象添加了自定义__del__方法，并在其中添加了一条print语句，然后将它们删除，那么为什么内存使用量从未下降?

谢谢

迪恩

解决方案

在大多数当前的OS上，动态内存的分配发生在通常称为堆的位置上(不要与同名的数据结构混淆)，从本质上讲，它只是一个连续的区域，从某个基地址开始，一直延伸到基地址加上堆的当前大小.进程从一小堆开始(通常只有几页)开始，并通过以系统页面大小的正整数倍为单位向上扩展该段来根据需要增长.在该空间内，分配/取消分配会创建较小的内存块以满足程序的需求，并且分配器需要使用任何元数据来跟踪分配的内存和未分配的内存.这些较小的内存块可能会随着时间的推移而重新分配，从而使堆段基本上为空，但是除非对其进行专门编码，否则堆段很少收缩.结果，一个长时间运行的进程似乎会出现一个堆段，该堆段的大小与峰值使用情况一样大.如果需要释放物理内存以供其他进程使用，则正在使用但不再可用的页面将倾向于移去交换，但是该进程映像似乎仍然是大"的，并且通常不会缩小无需重新启动.可以通过其他方式使用其他内存分配机制来解决该问题(例如，将临时文件映射为内存空间，并在完成后取消映射并删除它)，但是需要对此程序进行特殊编码.使用标准库分配例程(分别为malloc/free或new/delete)以C或C ++编写的应用程序，包括Python编译器/解释器，往往会表现出上述行为.

I am writing a twisted PB application that seems to use a very large amount of memory that never seems to get released when the user disconnects.

I have a pb.Root object that clients connect to and invoke a remote method that returns a pb.Referenceable object this object reads a large amount of information into memory (about 2GBs worth of data) when it is created to speed up the actions. This object, along with some other information about the client is inserted into a list.

When the client disconnects from the server, I invoke some clean up actions on this object to remove references to the cache object that is being stored. The chunkCache is the dict that i am storing the data in.

def disconnected(self):
    self.connected = False
    self.chunkCache = None
    self.cur.close()

Once the Client has disconnected the memory usage according to top never drops it still says 2Gb.

Should I be worried about this or will the allocated memory get released when its needed, or if not any ideas how I can release this memory? It is created when the object is created and not passed anywhere else.

Inside that object I do have one deferToThread call, could this be stopping the item being released?

I'm using python 2.7 on Linux.

UPDATE:

Im confused, I have just added custom __del__ methods to my object and put a print statement in there and they are being deleted, so why does the memory usage never drop?

Thanks

Dean

解决方案

On most current OSes, allocation of dynamic memory happens on what is often called the heap (not to be confused with the data structure of the same name), which is essentially just a contiguous region that starts at some base address and extends to the base address plus the current size of the heap. Processes start out with a small heap - typically just a few pages - and it grows as needed by extending the segment upwards in units of positive integer multiples of the system page size. Within that space, the allocation/deallocation create smaller chunks of memory to satisfy the needs of the program, along with whatever metadata the allocator needs to track what memory is allocated and what isn't. These smaller pieces of memory may be deallocated over time, leaving the heap segment essentially empty, but unless it's specifically coded, the heap segment is rarely if ever shrunk. As a result, a long-running process will tend to appear to have a heap segment that is as large as it's peak usage has been. The pages that were in use but aren't any more will tend to be moved off to swap if physical memory needs to be freed up for other processes, but the process image still appears to be "large", and generally won't shrink without being restarted. There are ways around that using other memory allocation mechanisms (e.g. mapping a temporary file for memory space and unmapping and removing it when done), but the program needs to be specifically coded to do so. Applications written in C or C++ that use the standard library allocation routines (malloc/free or new/delete, respectively), including the Python compiler/interpreter, will tend to exhibit the above behavior.

这篇关于Python内存使用量没有下降的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！