列出 ipython 和 jupyter 中的内存使用情况 [英] list memory usage in ipython and jupyter

问题描述

我有几个(几乎十个)Gb 的内存被 ipython 内核占用.我认为这来自我可能在某些操作期间产生的大对象(矩阵、列表、numpy 数组，...)，现在我不再需要了.

I have a few (almost ten) Gb of memory taken by the ipython kernel. I think this is coming from large objects (matrices, lists, numpy arrays, ...) that I might have produced during some operation and now I do not need anymore.

我想列出我定义的所有对象并按它们的内存占用量对它们进行排序.有没有一种简单的方法可以做到这一点?对于某些类型，有 nbytes 方法，但不是所有的......所以我正在寻找一种通用的方法来列出我制作的所有对象及其内存占用.

I would like to list all of the objects I have defined and sort them by their memory footprint. Is there a simple way to do that? For certain types there is nbytes method, but not for all ... so I am looking for a general way to list all objects I have made and their memory occupation.

推荐答案

假设您使用的是 ipython 或 jupyter，您将需要做一些工作获取您定义的所有对象的列表.这意味着获取 globals() 中可用的所有内容并过滤掉 modules、builtins、ipython 对象 的对象，等等.一旦你确定你有这些对象，那么你可以继续使用 sys.getsizeof 获取它们的大小.总结如下:

Assuming that you are using ipython or jupyter, you will need to do a little bit of work to get a list all of the objects you have defined. That means taking everything available in globals() and filtering out objects that are modules, builtins, ipython objects, etc. Once you are sure you have those objects, then you can proceed to grabbing their sizes with sys.getsizeof. This can be summed up as follows:

import sys

# These are the usual ipython objects, including this one you are creating
ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']

# Get a sorted list of the objects and their sizes
sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)

请记住，对于 python 对象(使用 python 的内置函数创建的对象)，sys.getsizeof 将非常准确.但是对于使用第三方库创建的对象，它可能有点不准确.此外，请注意，如果对象由垃圾收集器管理，sys.getsizeof 会增加额外的垃圾收集器开销.所以，有些东西看起来可能比实际重一些.

Please keep in mind that for python objects (those created with python's builtin functions), sys.getsizeof will be very accurate. But it can be a bit inaccurate on objects created using third-party libraries. Furthermore, please be mindful that sys.getsizeof adds an additional garbage collector overhead if the object is managed by the garbage collector. So, some things may look a bit heavier than they actually are.

作为旁注，numpy 的 .nbytes 方法可能有些误导，因为它不包括数组对象的非元素属性消耗的内存.

As a side note, numpy's .nbytes method can be somewhat misleading in that it does not include memory consumed by non-element attributes of the array object.

我希望这会有所帮助.

这篇关于列出 ipython 和 jupyter 中的内存使用情况的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！