python - 内存没有返回给内核 [英] python - memory not being given back to kernel

问题描述

我有一个非常简单的分配内存的脚本，dels 是对一个相当大的对象的唯一引用，同时打印 heapy 和 pidstat报告.运行脚本后，heapy 告诉我不应该使用太多内存，而 pidstat 告诉我相反:

from guppy import hpy导入时间导入系统导入操作系统'''1) 在开始之后和实际做任何工作之前打印 heapy 和 pidstat 报告2)在一个简单的二维数组中分配一些内存3) 打印 heapy 和 pidstat 报告4) 删除 d2 数组(尝试进行垃圾回收)5) 打印 heapy 和 pidstat 报告6) 睡眠以便 pidstat 可以继续运行以检查内存'''def pidstat(msg):打印 '================================打印消息os.system('pidstat -r -p %s' % os.getpid())打印 '++++++++++++++++++++++++++++++++'打印 hpy().heap()[0]打印 '================================pidstat('在做任何事情之前')文档 = []对于范围 (0, 10000) 中的文档:docs.append([j for j in range(0, 1000)])pidstat('将所有文档提取到内存中后')删除文档pidstat('释放文档后')时间.睡眠(60)

输出如下:

<前>================================在做任何事情之前Linux 2.6.38-15-generic (hersheezy) 08/14/2012 _x86_64_ (4 CPU)01:05:20 PM PID minflt/s majflt/s VSZ RSS %MEM 命令01:05:20 PM 5360 0.44 0.00 44768 9180 0.11 蟒蛇+++++++++++++++++++++++++++++++++++一组 19760 个对象的分区.总大小 = 1591024 字节.索引计数 % 大小 % 累积 % 种类(类/类的字典)0 19760 100 1591024 100 1591024 100 字符串================================================================将所有文档提取到内存中后Linux 2.6.38-15-generic (hersheezy) 08/14/2012 _x86_64_ (4 CPU)01:05:21 PM PID minflt/s majflt/s VSZ RSS %MEM 命令01:05:21 PM 5360 8.95 0.00 318656 279120 3.49 蟒蛇+++++++++++++++++++++++++++++++++++一组 7431665 个对象的分区.总大小 = 178359960 字节.索引计数 % 大小 % 累积 % 种类(类/类的字典)0 7431665 100 178359960 100 178359960 100 国际================================================================释放文档后Linux 2.6.38-15-generic (hersheezy) 08/14/2012 _x86_64_ (4 CPU)01:05:29 PM PID minflt/s majflt/s VSZ RSS %MEM 命令01:05:29 PM 5360 40.23 0.00 499984 460480 5.77 蟒蛇+++++++++++++++++++++++++++++++++++一组 19599 个对象的分区.总大小 = 1582016 字节.索引计数 % 大小 % 累积 % 种类(类/类的字典)0 19599 100 1582016 100 1582016 100 条================================

如何确保将这些内存返回给操作系统?

解决方案

内存在 python 进程内可用于重用的时间和释放到操作系统的时间可能存在差异.特别是，标准 Python 解释器 (CPython) 为特定类型的对象维护自己的池和空闲列表.它将自己重用这些池中的内存，但一旦使用过就不会将其释放给操作系统.

参见 这个了解更多详情.

I have a very simple script that allocates memory, dels the only reference to a sizable object, all the while printing heapy and pidstat reports. After running the script, heapy tells me that there should not be much memory being used while pidstat tells me the opposite:

from guppy import hpy
import time
import sys
import os

'''
1) print heapy and pidstat report after starting and before actually doing any work
2) allocate some memory in a simple 2d array
3) print heapy and pidstat report
4) del the d2 array (attempt at garbage collection)
5) print heapy and pidstat report
6) sleep so pidstat can continue to be run to check on memory
'''

def pidstat(msg):
    print '==============================='
    print msg
    os.system('pidstat -r -p %s' % os.getpid())
    print '+++++++++++++++++++++++++++++++'
    print hpy().heap()[0]
    print '==============================='

pidstat('before doing anything')
docs = []
for doc in range(0, 10000):
    docs.append([j for j in range(0, 1000)])

pidstat('after fetching all the docs into memory')
del docs

pidstat('after freeing the docs')
time.sleep(60)

The output looks as follows:

===============================
before doing anything
Linux 2.6.38-15-generic (hersheezy)     08/14/2012  _x86_64_    (4 CPU)

01:05:20 PM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
01:05:20 PM      5360      0.44      0.00   44768   9180   0.11  python
+++++++++++++++++++++++++++++++
Partition of a set of 19760 objects. Total size = 1591024 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  19760 100  1591024 100   1591024 100 str
===============================
===============================
after fetching all the docs into memory
Linux 2.6.38-15-generic (hersheezy)     08/14/2012  _x86_64_    (4 CPU)

01:05:21 PM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
01:05:21 PM      5360      8.95      0.00  318656 279120   3.49  python
+++++++++++++++++++++++++++++++
Partition of a set of 7431665 objects. Total size = 178359960 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 7431665 100 178359960 100 178359960 100 int
===============================
===============================
after freeing the docs
Linux 2.6.38-15-generic (hersheezy)     08/14/2012  _x86_64_    (4 CPU)

01:05:29 PM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
01:05:29 PM      5360     40.23      0.00  499984 460480   5.77  python
+++++++++++++++++++++++++++++++
Partition of a set of 19599 objects. Total size = 1582016 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  19599 100  1582016 100   1582016 100 str
===============================

How can I make sure this memory is returned to the operating system?

解决方案

There can be a difference between when memory is made available for reuse inside the python process and when it is released to the OS. In particular, the standard Python interpreter (CPython) maintains its own pools and free lists for particular kinds of objects. It will reuse memory in these pools itself, but never releases it to the OS once it's been used.

See this for more details.

这篇关于python - 内存没有返回给内核的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！