Python中(命名的)元组的字典和速度/RAM性能 [英] Dictionary of (named) tuples in Python and speed/RAM performance

问题描述

我正在创建一百万个元组项的字典 d ，理想情况下，我想通过以下方式访问它们:

I'm creating a dictionary d of one million of items which are tuples, and ideally I'd like to access them with:

d[1634].id       # or  d[1634]['id']
d[1634].name     # or  d[1634]['name']
d[1634].isvalid  # or  d[1634]['isvalid']

而不是 d [1634] [0] ， d [1634] [1] ， d [1634] [2] 不太明确.

根据我的测试:

import os, psutil, time, collections, typing
Tri = collections.namedtuple('Tri', 'id,name,isvalid')
Tri2 = typing.NamedTuple("Tri2", [('id', int), ('name', str), ('isvalid', bool)])
t0 = time.time()
# uncomment only one of these 4 next lines:
d = {i: (i+1, 'hello', True) for i in range(1000000)}                                 # tuple
# d = {i: {'id': i+1, 'name': 'hello', 'isvalid': True} for i in range(1000000)}      # dict
# d = {i: Tri(id=i+1, name='hello', isvalid=True) for i in range(1000000)}            # namedtuple
# d = {i: Tri2(id=i+1, name='hello', isvalid=True) for i in range(1000000)}            # NamedTuple
print('%.3f s  %.1f MB' % (time.time()-t0, psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2))

"""
tuple:       0.257 s  193.3 MB
dict:        0.329 s  363.6 MB
namedtuple:  1.253 s  193.3 MB  (collections)
NamedTuple:  1.250 s  193.5 MB  (typing)
"""

与 tuple 相比，
• 使用 dict 可使RAM使用量增加一倍与 tuple 相比，
• 使用 namedtuple 或 NamedTuple 所花费的时间乘以5！
  • using a dict doubles the RAM usage, compared to a tuple
  • using a namedtuple or NamedTuple multiplies by 5 the time spent, compared to a tuple!

  问题:Python 3中是否有类似元组的数据结构，该结构允许使用 x.id ， x.name 等访问数据.以及RAM和CPU效率高吗?

  Question: is there a tuple-like data structure in Python 3 which allows to access the data with x.id, x.name, etc. and also is RAM and CPU efficient?

  注意:

  • 在我的实际用例中， tuple 类似于类型为(uint64，uint64，bool)的C结构.

  • in my real use case, the tuple is something like a C-struct of type (uint64, uint64, bool).

  我也尝试过:

  • 插槽(为避免内部对象的 __ dict __ ，请参见

  • slots (to avoid the interal object's __dict__, see Usage of __slots__?)

  数据类:

  @dataclasses.dataclass
  class Tri3:
      id: int
      ...
  

  • ctypes.Structure :

    class Tri7(ctypes.Structure):
        _fields_ = [("id", ctypes.c_int), ...]
    

  但效果并不好(所有人都约1.2秒)，就性能而言，没有什么比真正的 tuple 更好的了.

  but it was not better (all of them ~ 1.2 sec.), nothing close to a genuine tuple in terms of performance

  还有其他选择: Python中类似C的结构

  推荐答案

  Cython的

  Cython's cdef-classes might be what you want: They use less memory than the pure Python classes, even if it comes at costs of more overhead when accessing members (because fields are stored as C-values and not Python-objects).

  例如:

  %%cython
  cdef class CTuple:
      cdef public unsigned long long int id
      cdef public str name
      cdef public bint isvalid
      
      def __init__(self, id, name, isvalid):
          self.id = id
          self.name = name
          self.isvalid = isvalid
  

  可按需使用:

  ob=CTuple(1,"mmm",3)
  ob.id, ob.name, ob.isvalid # prints (2, "mmm", 3)
  

  时间/内存消耗

  首先，我的计算机上的基准:

  First, the baseline on my machine:

  0.258 s  252.4 MB  # tuples
  0.343 s  417.5 MB  # dict
  1.181 s  264.0 MB  # namedtuple collections
  

  使用 CTuple ，我们得到:

  0.306 s  191.0 MB
  

  这几乎一样快，并且需要更少的内存.

  which is almost as fast and needs considerable less memory.

  如果在编译时不清楚C型成员，则可以使用简单的python对象:

  If the C-type of members isn't clear at compile time, one could use simple python-objects:

  %%cython
  cdef class PTuple:
      cdef public object id
      cdef public object name
      cdef public object isvalid
      
      def __init__(self, id, name, isvalid):
          self.id = id
          self.name = name
          self.isvalid = isvalid
  

  时间有点令人惊讶:

  0.648 s  249.8 MB
  

  我没想到它会比 CTuple -version慢得多，但是至少它快两倍于命名元组.

  I didn't expect it to be so much slower than the CTuple-version, but at least it is twice as fast as named tuples.

  此方法的一个缺点是它需要编译.但是，Cython提供了 cython.inline ，可用于编译动态创建的Cython代码.

  One disadvantage of this approach is that it needs compilation. Cython however offers cython.inline which can be used to compile Cython-code created on-the-fly.

  我发布了 cynamedtuple ，可以通过 pip install cynamedtuple ，并且基于以下原型:

  I've released cynamedtuple which can be installed via pip install cynamedtuple, and is based on the prototype bellow:

  import cython
  
  # for generation of cython code:
  tab = "    "
  def create_members_definition(name_to_ctype):
      members = []
      for my_name, my_ctype in name_to_ctype.items():
          members.append(tab+"cdef public "+my_ctype+" "+my_name)
      return members
  
  def create_signature(names):
      return tab + "def __init__(self,"+", ".join(names)+"):"
  
  def create_initialization(names):
      inits = [tab+tab+"self."+x+" = "+x for x in names]
      return inits
  
  def create_cdef_class_code(classname, names):
      code_lines = ["cdef class " + classname + ":"]
      code_lines.extend(create_members_definition(names))
      code_lines.append(create_signature(names.keys()))
      code_lines.extend(create_initialization(names.keys()))
      return "\n".join(code_lines)+"\n"
  
  # utilize cython.inline to generate and load pyx-module:
  def create_cnamedtuple_class(classname, names):
      code = create_cdef_class_code(classname, names)
      code = code + "GenericClass = " + classname +"\n"
      ret = cython.inline(code)
      return ret["GenericClass"]
  

  可以如下使用，以从上方动态定义 CTuple :

  Which can be used as follows, to dynamically define CTuple from above:

  CTuple = create_cnamedtuple_class("CTuple", 
                                   {"id":"unsigned long long int", 
                                    "name":"str",
                                    "isvalid":"bint"})
  
  ob = CTuple(1,"mmm",3)
  ... 
  

  ---

  另一种替代方法是使用jit编译和Numba的 jitted-类提供了这种可能性.但是，它们似乎要慢得多:

  ---

  Another alternative could be to use jit-compilation and Numba's jitted-classes which offer this possibility. They however seem to be much slower:

  from numba import jitclass, types
  
  spec = [
      ('id', types.uint64), 
      ('name', types.string),
      ('isvalid',  types.uint8),
  ]
  
  @jitclass(spec)
  class NBTuple(object):
      def __init__(self, id, name, isvalid):
          self.id = id
          self.name = name
          self.isvalid = isvalid
  

  结果为:

  20.622 s  394.0 MB
  

  因此，numba入门课程不是一个不错的选择.

  so numba jitted classes are not (yet?) a good choice.

  这篇关于Python中(命名的)元组的字典和速度/RAM性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！