大约80000个项目的列表在python中会消耗多少内存? [英] Approximately how much memory would a list of 80000 items consume in python?

问题描述

我有一个python列表，其中包含80000个列表.这些内部列表中的每一个或多或少都具有以下格式:

I have a python list, which consists of 80000 lists. Each of these inner lists more or less have this format:

["012345", "MYNAME" "Mon", "A", 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20]

您能告诉我们这个由80000个列表组成的列表消耗多少内存吗?

Could you tell approximately how much memory would this list consisting of 80000 lists consume?

在python中很大的列表上使用和操作是否常见/确定?我要做的大多数操作是使用列表理解方法从此列表中提取数据.

And is it common/OK to use and operate on lists that big in python? Most of the operations I do is to extract data from this list with list comprehension method.

实际上，我想学习的是:python足够快，可以使用列表理解方法从大列表中提取数据.我希望我的脚本很快

Actually, what I would like to learn is: is python fast enough to extract data from that big lists using list comprehension methods. I want my script to be fast

推荐答案

In [39]: lis=["012345", "MYNAME" "Mon", "A", 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20,
     20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20]

In [40]: k=[lis[:] for _ in xrange(80000)]

In [41]: k.__sizeof__()
Out[41]: 325664

In [42]: sys.getsizeof(k)  #after gc_head
Out[42]: 325676

按照 sysmodule.c 中的代码看起来它调用了__sizeof__方法来获取对象的大小.

As per the code in sysmodule.c it looks like it calls __sizeof__ method to get the size of an object.

   837   method = _PyObject_LookupSpecial(o, &PyId___sizeof__);   
   838     if (method == NULL) {
   839         if (!PyErr_Occurred())
   840             PyErr_Format(PyExc_TypeError,
   841                          "Type %.100s doesn't define __sizeof__",
   842                          Py_TYPE(o)->tp_name);
   843     }
   844     else {
   845         res = PyObject_CallFunctionObjArgs(method, NULL);
   846         Py_DECREF(method);
   847     }

，然后增加一些gc开销:

   860     /* add gc_head size */
   861     if (PyObject_IS_GC(o)) {
   862         PyObject *tmp = res;
   863         res = PyNumber_Add(tmp, gc_head_size);
   864         Py_DECREF(tmp);
   865     }
   866     return res;
   867 }

我们还可以按照

We can also use the recursive sizeof recipe as suggested in docs to recursively calculate the size of each container:

In [17]: total_size(k)  #from recursive sizeof recipe
Out[17]: 13125767

In [18]: sum(y.__sizeof__() for x in k for y in x)
Out[18]: 34160000

这篇关于大约80000个项目的列表在python中会消耗多少内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！