列出ipython和jupyter中的内存使用情况 [英] list memory usage in ipython and jupyter

问题描述

我的ipython内核占用了几(近十)Gb的内存.我认为这是来自一些操作期间可能产生的大对象(矩阵，列表，numpy数组等)，现在不再需要.

I have a few (almost ten) Gb of memory taken by the ipython kernel. I think this is coming from large objects (matrices, lists, numpy arrays, ...) that I might have produced during some operation and now I do not need anymore.

我想列出我定义的所有对象，并按其内存占用量对其进行排序.有没有简单的方法可以做到这一点?对于某些类型，可以使用nbytes方法，但不能对所有类型都使用... c，因此我正在寻找一种通用的方法来列出我制作的所有对象及其内存占用情况.

I would like to list all of the objects I have defined and sort them by their memory footprint. Is there a simple way to do that? For certain types there is nbytes method, but not for all ... so I am looking for a general way to list all objects I have made and their memory occupation.

推荐答案

假设您正在使用ipython或jupyter，则需要做一些工作来获取所有对象的列表 您 已定义.这意味着要使用globals()中的所有内容，并过滤出modules，builtins，ipython objects等对象.一旦确定拥有这些对象，则可以继续使用sys.getsizeof来获取它们的大小. .可以总结如下:

Assuming that you are using ipython or jupyter, you will need to do a little bit of work to get a list all of the objects you have defined. That means taking everything available in globals() and filtering out objects that are modules, builtins, ipython objects, etc. Once you are sure you have those objects, then you can proceed to grabbing their sizes with sys.getsizeof. This can be summed up as follows:

import sys

# These are the usual ipython objects, including this one you are creating
ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']

# Get a sorted list of the objects and their sizes
sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)

请记住，对于python对象(使用python内置函数创建的对象)，sys.getsizeof将非常准确.但是，使用第三方库创建的对象可能有点不准确.此外，请注意，如果对象由垃圾收集器管理，则sys.getsizeof会增加额外的垃圾收集器开销.因此，有些事情看起来可能比实际重.

Please keep in mind that for python objects (those created with python's builtin functions), sys.getsizeof will be very accurate. But it can be a bit inaccurate on objects created using third-party libraries. Furthermore, please be mindful that sys.getsizeof adds an additional garbage collector overhead if the object is managed by the garbage collector. So, some things may look a bit heavier than they actually are.

请注意，numpy的.nbytes方法可能会引起误解，因为它不包含数组对象的非元素属性消耗的内存.

As a side note, numpy's .nbytes method can be somewhat misleading in that it does not include memory consumed by non-element attributes of the array object.

我希望这会有所帮助.

这篇关于列出ipython和jupyter中的内存使用情况的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！