Python 中最快(访问)的类结构对象是什么? [英] What is the fastest (to access) struct-like object in Python?

问题描述

我正在优化一些代码，其主要瓶颈在于运行和访问非常大的类结构对象列表.目前我正在使用命名元组，以提高可读性.但是使用timeit"的一些快速基准测试表明，在性能是一个因素的情况下，这确实是错误的方法:

带有 a、b、c 的命名元组:

<预><代码>>>>timeit("z = a.c", "from __main__ import a")0.38655471766332994

使用__slots__的类，带有a、b、c:

<预><代码>>>>timeit("z = b.c", "from __main__ import b")0.14527461047146062

带有键 a、b、c 的字典:

<预><代码>>>>timeit("z = c['c']", "from __main__ import c")0.11588272541098377

具有三个值的元组，使用一个常量键:

<预><代码>>>>timeit("z = d[2]", "from __main__ import d")0.11106188992948773

使用常量键列出三个值:

<预><代码>>>>timeit("z = e[2]", "from __main__ import e")0.086038238242508669

具有三个值的元组，使用本地键:

<预><代码>>>>timeit("z = d[key]", "from __main__ import d, key")0.11187358437882722

使用本地键列出三个值:

<预><代码>>>>timeit("z = e[key]", "from __main__ import e, key")0.088604143037173344

首先，这些小的 timeit 测试是否会导致它们无效?我每次都跑了几次，以确保没有随机系统事件将它们扔掉，结果几乎相同.

字典似乎在性能和可读性之间提供了最佳平衡，类排在第二位.这是不幸的，因为就我的目的而言，我还需要对象类似于序列；因此我选择了命名元组.

列表要快得多，但常量键不可维护；我必须创建一堆索引常量，即 KEY_1 = 1、KEY_2 = 2 等，这也不理想.

我是否坚持这些选择，还是我错过了其他选择?

解决方案

需要牢记的一点是，namedtuples 是为作为元组访问而优化的.如果您将访问器更改为 a[2] 而不是 a.c，您将看到与元组相似的性能.原因是名称访问器有效地转换为对 self[idx] 的调用，因此要同时支付索引和名称查找价格.

如果您的使用模式是按名称访问很常见，但作为元组访问则不常见，您可以编写一个快速等效的 namedtuple 以相反的方式执行操作:将索引查找推迟到按名称访问.但是，您将为索引查找付出代价.例如，这是一个快速实现:

def makestruct(name, fields):字段 = fields.split()导入文本换行模板 = textwrap.dedent("""类{名称}(对象):__slots__ = {fields!r}def __init__(self, {args}):{self_fields} = {args}def __getitem__(self, idx):返回 getattr(self, fields[idx])).格式(姓名=姓名，字段=字段，args=','.join(fields),self_fields=','.join('self.' + f for f in fields))d = {'字段':字段}d 中的 exec 模板返回 d[名称]

但是当必须调用 __getitem__ 时，时机非常糟糕:

namedtuple.a : 0.473686933517命名元组[0]:0.180409193039结构.a:0.180846214294结构[0]:1.32191514969

即，与用于属性访问的 __slots__ 类相同的性能(不出所料 - 就是这样)，但由于基于索引的访问中的双重查找而造成巨大损失.(值得注意的是 __slots__ 实际上在速度方面并没有多大帮助.它节省了内存，但没有它们，访问时间大致相同.)

第三种选择是复制数据，例如.从列表子类并将值存储在属性和列表数据中.但是，您实际上并没有获得与列表等效的性能.仅仅在子类化(引入对纯 python 重载的检查)中就有很大的速度损失.因此在这种情况下 struct[0] 仍然需要大约 0.5 秒(与原始列表的 0.18 相比)，并且您的内存使用量增加了一倍，因此这可能不值得.

I'm optimizing some code whose main bottleneck is running through and accessing a very large list of struct-like objects. Currently I'm using namedtuples, for readability. But some quick benchmarking using 'timeit' shows that this is really the wrong way to go where performance is a factor:

Named tuple with a, b, c:

>>> timeit("z = a.c", "from __main__ import a")
0.38655471766332994

Class using __slots__, with a, b, c:

>>> timeit("z = b.c", "from __main__ import b")
0.14527461047146062

Dictionary with keys a, b, c:

>>> timeit("z = c['c']", "from __main__ import c")
0.11588272541098377

Tuple with three values, using a constant key:

>>> timeit("z = d[2]", "from __main__ import d")
0.11106188992948773

List with three values, using a constant key:

>>> timeit("z = e[2]", "from __main__ import e")
0.086038238242508669

Tuple with three values, using a local key:

>>> timeit("z = d[key]", "from __main__ import d, key")
0.11187358437882722

List with three values, using a local key:

>>> timeit("z = e[key]", "from __main__ import e, key")
0.088604143037173344

First of all, is there anything about these little timeit tests that would render them invalid? I ran each several times, to make sure no random system event had thrown them off, and the results were almost identical.

It would appear that dictionaries offer the best balance between performance and readability, with classes coming in second. This is unfortunate, since, for my purposes, I also need the object to be sequence-like; hence my choice of namedtuple.

Lists are substantially faster, but constant keys are unmaintainable; I'd have to create a bunch of index-constants, i.e. KEY_1 = 1, KEY_2 = 2, etc. which is also not ideal.

Am I stuck with these choices, or is there an alternative that I've missed?

解决方案

One thing to bear in mind is that namedtuples are optimised for access as tuples. If you change your accessor to be a[2] instead of a.c, you'll see similar performance to the tuples. The reason is that the name accessors are effectively translating into calls to self[idx], so pay both the indexing and the name lookup price.

If your usage pattern is such that access by name is common, but access as tuple isn't, you could write a quick equivalent to namedtuple that does things the opposite way: defers index lookups to access by-name. However, you'll pay the price on the index lookups then. Eg here's a quick implementation:

def makestruct(name, fields):
    fields = fields.split()
    import textwrap
    template = textwrap.dedent("""
    class {name}(object):
        __slots__ = {fields!r}
        def __init__(self, {args}):
            {self_fields} = {args}
        def __getitem__(self, idx): 
            return getattr(self, fields[idx])
    """).format(
        name=name,
        fields=fields,
        args=','.join(fields), 
        self_fields=','.join('self.' + f for f in fields))
    d = {'fields': fields}
    exec template in d
    return d[name]

But the timings are very bad when __getitem__ must be called:

namedtuple.a  :  0.473686933517 
namedtuple[0] :  0.180409193039
struct.a      :  0.180846214294
struct[0]     :  1.32191514969

ie, the same performance as a __slots__ class for attribute access (unsurprisingly - that's what it is), but huge penalties due to the double lookup in index-based accesses. (Noteworthy is that __slots__ doesn't actually help much speed-wise. It saves memory, but the access time is about the same without them.)

One third option would be to duplicate the data, eg. subclass from list and store the values both in the attributes and listdata. However you don't actually get list-equivalent performance. There's a big speed hit just in having subclassed (bringing in checks for pure-python overloads). Thus struct[0] still takes around 0.5s (compared with 0.18 for raw list) in this case, and you do double the memory usage, so this may not be worth it.

这篇关于Python 中最快(访问)的类结构对象是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！