什么是Python中最快（访问）结构体对象？ [英] What is the fastest (to access) struct-like object in Python?

问题描述

我正在优化一些主要瓶颈正在运行的代码，并访问一个非常大的结构体对象列表。目前我正在使用namedtuples，因为可读性。但是使用'timeit'的一些快速基准测试表明，这是一个非常错误的方式去执行性能的一个因素：

使用a，b，c： p>

 >>> timeit（z = ac，from __main__ import a）
 0.38655471766332994 
  

使用 __ slots __ 与a，b，c：

 >> ;> timeit（z = bc，__main__ import b）
 0.14527461047146062 
  

用键a，b，c：

 >>> timeit（z = c ['c']，from __main__ import c）
 0.11588272541098377 
  

具有三个值的元组，使用常量键：

 >>> timeit（z = d [2]，from __main__ import d）
 0.11106188992948773 
  

列出三个值，使用常量键：

 >>> timeit（z = e [2]，from __main__ import e）
 0.086038238242508669 
  

具有三个值的元组，使用本地键：

 >>> timeit（z = d [key]，from __main__ import d，key）
 0.11187358437882722 
  

使用本地密钥列出三个值：

 >>> timeit（z = e [key]，from __main__ import e，key）
 0.088604143037173344 
  

首先，有没有关于这些小的 timeit 测试会使它们无效的东西？我运行了几次，以确保没有随机的系统事件被抛弃，结果几乎相同。

似乎字典提供了最佳的平衡表现和可读性，班级排在第二。这是不幸的，因为为了我的目的，我也需要这个对象是顺序的;因此我选择了namedtuple。

列表速度快得多，但常数键不可维护;我必须创建一堆索引常数，即KEY_1 = 1，KEY_2 = 2等，这也不是很理想。

我坚持这些选择，还是有一个我错过的替代方案？

解决方案

需要记住的是，namedtuples是针对作为元组访问如果您将访问者更改为 a [2] 而不是 ac ，则会看到类似于元组的性能。原因是名称访问者正在有效地转换为调用self [idx]，所以支付索引和的名称查找价格。

如果您的使用模式是这样访问的名称是常见的，但访问作为元组不是，您可以/ em>写一个快速相当于namedtuple做相反的方式：将索引查找延迟访问-名称。然而，您将在索引查找中支付价格。例如，这是一个快速实现：

  def makestruct（name，fields）：
 fields = fields.split（）
 import textwrap 
 template = textwrap.dedent（\ 
 class {name}（object）：
 __slots__ = {fields！r} 
 def __init __ self，{args}）：
 {self_fields} = {args} 
 def __getitem __（self，idx）：
 return getattr（self，fields [idx]）
 ）.format（
 name = name，
 fields = fields，
 args ='，'。join（fields），
 self_fields ='，'。 。'+ f for f in fields））
d = {'fields'：fields} 
 exec template in d 
 return d [name] 
  

但是当 __ getitem __ 必须调用时，时间是非常糟糕的：

  namedtuple.a：0.473686933517 
 namedtuple [0]：0.180409193039 
 struct.a：0.180846214294 
 struct [0]： 1.32191514969 
  
 
即，与 __ slots __ 类的属性访问相同（不出意外 - 这是什么），但由于在基于索引的访问中的双重查找而造成的巨大惩罚。 （值得注意的是，$ code> __ slots __ 实际上并没有帮助太多速度，它可以节省内存，但访问时间大致相同。）
 
 
 
第三个选项是复制数据，例如。从列表中的子类，并存储属性和列表数据中的值。但是实际上并没有获得与列表相当的性能。在子类中有一个很大的速度（引入检查纯粹的python重载）。因此，在这种情况下，struct [0]仍然需要大约0.5s（与原始列表相比为0.18），并且您的内存使用倍增，所以这可能不值得。
 
I'm optimizing some code whose main bottleneck is running through and accessing a very large list of struct-like objects.  Currently I'm using namedtuples, for readability.  But some quick benchmarking using 'timeit' shows that this is really the wrong way to go where performance is a factor:

Named tuple with a, b, c:
>>> timeit("z = a.c", "from __main__ import a")
0.38655471766332994
Class using __slots__, with a, b, c:
>>> timeit("z = b.c", "from __main__ import b")
0.14527461047146062
Dictionary with keys a, b, c:
>>> timeit("z = c['c']", "from __main__ import c")
0.11588272541098377
Tuple with three values, using a constant key:
>>> timeit("z = d[2]", "from __main__ import d")
0.11106188992948773
List with three values, using a constant key:
>>> timeit("z = e[2]", "from __main__ import e")
0.086038238242508669
Tuple with three values, using a local key:
>>> timeit("z = d[key]", "from __main__ import d, key")
0.11187358437882722
List with three values, using a local key:
>>> timeit("z = e[key]", "from __main__ import e, key")
0.088604143037173344
First of all, is there anything about these little timeit tests that would render them invalid?  I ran each several times, to make sure no random system event had thrown them off, and the results were almost identical.

It would appear that dictionaries offer the best balance between performance and readability, with classes coming in second.  This is unfortunate, since, for my purposes, I also need the object to be sequence-like; hence my choice of namedtuple.

Lists are substantially faster, but constant keys are unmaintainable; I'd have to create a bunch of index-constants, i.e. KEY_1 = 1, KEY_2 = 2, etc. which is also not ideal.

Am I stuck with these choices, or is there an alternative that I've missed?
解决方案
One thing to bear in mind is that namedtuples are optimised for access as tuples.  If you change your accessor to be a[2] instead of a.c, you'll see similar performance to the tuples.  The reason is that the name accessors are effectively translating into calls to self[idx], so pay both the indexing and the name lookup price.

If your usage pattern is such that access by name is common, but access as tuple isn't, you could write a quick equivalent to namedtuple that does things the opposite way: defers index lookups to access by-name.  However, you'll pay the price on the index lookups then.  Eg here's a quick implementation:
def makestruct(name, fields):
    fields = fields.split()
    import textwrap
    template = textwrap.dedent("""\
    class {name}(object):
        __slots__ = {fields!r}
        def __init__(self, {args}):
            {self_fields} = {args}
        def __getitem__(self, idx): 
            return getattr(self, fields[idx])
    """).format(
        name=name,
        fields=fields,
        args=','.join(fields), 
        self_fields=','.join('self.' + f for f in fields))
    d = {'fields': fields}
    exec template in d
    return d[name]
But the timings are very bad when __getitem__ must be called:
namedtuple.a  :  0.473686933517 
namedtuple[0] :  0.180409193039
struct.a      :  0.180846214294
struct[0]     :  1.32191514969
ie, the same performance as a __slots__ class for attribute access (unsurprisingly - that's what it is), but huge penalties due to the double lookup in index-based accesses.  (Noteworthy is that __slots__ doesn't actually help much speed-wise.  It saves memory, but the access time is about the same without them.)

One third option would be to duplicate the data, eg. subclass from list and store the values both in the attributes and listdata.  However you don't actually get list-equivalent performance.  There's a big speed hit just in having subclassed (bringing in checks for pure-python overloads).  Thus struct[0] still takes around 0.5s (compared with 0.18 for raw list) in this case, and you do double the memory usage, so this may not be worth it.

                        
这篇关于什么是Python中最快（访问）结构体对象？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！