在 Python 中释放内存 [英] Releasing memory in Python
问题描述
在以下示例中，我有一些关于内存使用的相关问题.
I have a few related questions regarding memory usage in the following example.
如果我在解释器中运行，
If I run in the interpreter,
foo = ['bar' for _ in xrange(10000000)]
我机器上使用的实际内存高达 80.9mb.然后我，
the real memory used on my machine goes up to 80.9mb. I then,
del foo
实际内存下降，但仅为 30.4mb.解释器使用 4.4mb 基线，那么不向操作系统释放 26mb 内存的优势是什么?是不是因为Python在提前计划"，认为你可能会再次使用那么多内存?
real memory goes down, but only to 30.4mb. The interpreter uses 4.4mb baseline so what is the advantage in not releasing 26mb of memory to the OS? Is it because Python is "planning ahead", thinking that you may use that much memory again?
为什么它特别释放 50.5mb - 释放的数量是基于什么?
Why does it release 50.5mb in particular - what is the amount that is released based on?
有没有办法强制 Python 释放所有已使用的内存(如果您知道不会再使用那么多内存)?
Is there a way to force Python to release all the memory that was used (if you know you won't be using that much memory again)?
注意这个问题不同于如何在 Python 中显式释放内存?因为这个问题主要涉及即使在解释器通过垃圾收集(使用 gc.collect 或不使用)释放对象之后，内存使用量从基线增加.
NOTE
This question is different from How can I explicitly free memory in Python?
because this question primarily deals with the increase of memory usage from baseline even after the interpreter has freed objects via garbage collection (with use of gc.collect or not).
推荐答案
在堆上分配的内存可能会受到高水位标记的影响.这被 Python 内部优化用于在 4 KiB 池中分配小对象 (PyObject_Malloc) 变得复杂，分配大小按 8 字节的倍数分类——最多 256 字节(3.3 中为 512 字节).池本身位于 256 KiB 的竞技场中，因此如果仅使用一个池中的一个区块，则不会释放整个 256 KiB 的竞技场.在 Python 3.3 中，小对象分配器被切换为使用匿名内存映射而不是堆，因此它应该在释放内存方面表现更好.
Memory allocated on the heap can be subject to high-water marks. This is complicated by Python's internal optimizations for allocating small objects (PyObject_Malloc) in 4 KiB pools, classed for allocation sizes at multiples of 8 bytes -- up to 256 bytes (512 bytes in 3.3). The pools themselves are in 256 KiB arenas, so if just one block in one pool is used, the entire 256 KiB arena will not be released. In Python 3.3 the small object allocator was switched to using anonymous memory maps instead of the heap, so it should perform better at releasing memory.
此外，内置类型维护先前分配的对象的空闲列表，这些对象可能会或可能不会使用小对象分配器.int 类型使用自己分配的内存维护一个空闲列表，清除它需要调用 PyInt_ClearFreeList().这可以通过执行完整的 gc.collect 间接调用.
Additionally, the built-in types maintain freelists of previously allocated objects that may or may not use the small object allocator. The int type maintains a freelist with its own allocated memory, and clearing it requires calling PyInt_ClearFreeList(). This can be called indirectly by doing a full gc.collect.
像这样尝试，然后告诉我你得到了什么.这是 psutil.Process.memory_info 的链接.
Try it like this, and tell me what you get. Here's the link for psutil.Process.memory_info.
import os
import gc
import psutil
proc = psutil.Process(os.getpid())
gc.collect()
mem0 = proc.memory_info().rss
# create approx. 10**7 int objects and pointers
foo = ['abc' for x in range(10**7)]
mem1 = proc.memory_info().rss
# unreference, including x == 9999999
del foo, x
mem2 = proc.memory_info().rss
# collect() calls PyInt_ClearFreeList()
# or use ctypes: pythonapi.PyInt_ClearFreeList()
gc.collect()
mem3 = proc.memory_info().rss
pd = lambda x2, x1: 100.0 * (x2 - x1) / mem0
print "Allocation: %0.2f%%" % pd(mem1, mem0)
print "Unreference: %0.2f%%" % pd(mem2, mem1)
print "Collect: %0.2f%%" % pd(mem3, mem2)
print "Overall: %0.2f%%" % pd(mem3, mem0)
输出:
Allocation: 3034.36%
Unreference: -752.39%
Collect: -2279.74%
Overall: 2.23%
我改用相对于进程 VM 大小的测量来消除系统中其他进程的影响.
I switched to measuring relative to the process VM size to eliminate the effects of other processes in the system.
当顶部的连续可用空间达到恒定、动态或可配置的阈值时，C 运行时(例如 glibc、msvcrt)会缩小堆.使用 glibc，您可以使用 mallopt (M_TRIM_THRESHOLD).鉴于此，与您释放的块相比，堆缩小更多(甚至更多)也就不足为奇了.
The C runtime (e.g. glibc, msvcrt) shrinks the heap when contiguous free space at the top reaches a constant, dynamic, or configurable threshold. With glibc you can tune this with mallopt (M_TRIM_THRESHOLD). Given this, it isn't surprising if the heap shrinks by more -- even a lot more -- than the block that you free.
在 3.x 中 range 不创建列表，因此上面的测试不会创建 1000 万个 int 对象.即使有，3.x 中的 int 类型基本上是一个 2.x long，它没有实现 freelist.
In 3.x range doesn't create a list, so the test above won't create 10 million int objects. Even if it did, the int type in 3.x is basically a 2.x long, which doesn't implement a freelist.
这篇关于在 Python 中释放内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
