库itertools与python代码相比的性能 [英] Performance of library itertools compared to python code

问题描述

作为对我的问题的回答找到两个列表相同的基于1的位置我得到了使用C库itertools加快速度的提示.

As answer to my question Find the 1 based position to which two lists are the same I got the hint to use the C-library itertools to speed up things.

要验证，我使用cProfile编写了以下测试代码:

To verify I coded the following test using cProfile:

from itertools import takewhile, izip

def match_iter(self, other):
    return sum(1 for x in takewhile(lambda x: x[0] == x[1],
                                        izip(self, other)))

def match_loop(self, other):
    element = -1
    for element in range(min(len(self), len(other))):
        if self[element] != other[element]:
            element -= 1
            break
    return element +1

def test():
    a = [0, 1, 2, 3, 4]
    b = [0, 1, 2, 3, 4, 0]

    print("match_loop a=%s, b=%s, result=%s" % (a, b, match_loop(a, b)))
    print("match_iter a=%s, b=%s, result=%s" % (a, b, match_iter(a, b)))

    i = 10000
    while i > 0:
        i -= 1
        match_loop(a, b)
        match_iter(a, b)

def profile_test():
    import cProfile
    cProfile.run('test()')

if __name__ == '__main__':
    profile_test()

match_iter()函数正在使用itertools，match_loop()函数是我在使用普通python之前实现的函数.

The function match_iter() is using the itertools and the function match_loop() is the one I had implemented before using plain python.

函数test()定义了两个列表，打印带有两个函数结果的列表以验证其是否正常工作.两个结果的期望值5都是列表的长度相等.然后在这两个函数上循环10,000次.

The function test() defines two lists, prints the lists with the results of the two functions to verify it is working. Both results have the expected value 5 which is the length for the lists being equal. Then it loops 10,000 times over the both functions.

最后，整个内容都使用profile_test()进行了分析.

Finally the whole thing is profiled using profile_test().

我了解到的是，izip没有在python3的itertools中实现，至少在我正在使用的debian wheezy whitch中没有实现.所以我已经使用python2.7进行了测试

What I learned than was that izip is not implemented in the itertools of python3, at least not in debian wheezy whitch I am using. So I had run the test with python2.7

以下是结果:

python2.7 match_test.py
match_loop a=[0, 1, 2, 3, 4], b=[0, 1, 2, 3, 4, 0], result=5
match_iter a=[0, 1, 2, 3, 4], b=[0, 1, 2, 3, 4, 0], result=5
         180021 function calls in 0.636 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.636    0.636 <string>:1(<module>)
        1    0.039    0.039    0.636    0.636 match_test.py:15(test)
    10001    0.048    0.000    0.434    0.000 match_test.py:3(match_iter)
    60006    0.188    0.000    0.275    0.000 match_test.py:4(<genexpr>)
    50005    0.087    0.000    0.087    0.000 match_test.py:4(<lambda>)
    10001    0.099    0.000    0.162    0.000 match_test.py:7(match_loop)
    20002    0.028    0.000    0.028    0.000 {len}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
    10001    0.018    0.000    0.018    0.000 {min}
    10001    0.018    0.000    0.018    0.000 {range}
    10001    0.111    0.000    0.387    0.000 {sum}

让我感到奇怪的是，看看cumtime值，我的普通python版本的10000个循环的值为0.162秒，而match_iter版本的值为0.434秒.

What makes me wonder is, looking at the cumtime values, my plain python version has a value of 0.162 seconds for 10,000 loops and the match_iter version takes 0.434 seconds.

一方面，python非常快，很棒，所以我不必担心.但这是否正确，C库完成工作的时间是简单python代码的两倍以上?还是我犯了致命错误?

For one thing python is very fast, great, so I do not have to worry. But can this be correct, that the C-library takes more than twice as long to finish the job as simple python code? Or am I doing a fatal mistake?

为验证我也使用python2.6进行了测试，这似乎更快，但循环和itertools之间的区别相同.

To verify I ran the test with python2.6 also, which seems to be even faster, but with the same difference between looping and itertools.

谁有经验并愿意提供帮助?

Who is experienced and willing to help?

推荐答案

我想这里的问题是您的测试列表很小-意味着任何差异都可能很小，并且创建迭代器的成本超过了它们的收益给予.

I imagine the issue here is your test lists are tiny - meaning any difference is likely to be minimal, and the cost of creating the iterators is outweighing the gains they give.

在较大的测试中(性能可能更重要)，使用sum()的版本可能会胜过其他版本.

In larger tests (where the performance is more likely to matter), the version using sum() will likely outperform the other version.

此外，还有样式问题-手动版本较长，并且依赖于按索引进行迭代，因此灵活性也较低.

Also, there is the matter of style - the manual version is longer, and relies on iterating by index, making it less flexible as well.

我认为最易读的解决方案是这样的:

I would argue the most readable solution would be something like this:

def while_equal(seq, other):
    for this, that in zip(seq, other):
        if this != that:
            return
        yield this

def match(seq, other):
    return sum(1 for _ in while_equal(seq, other))

有趣的是，在我的系统上对此稍作修改:

Interestingly, on my system a slightly modified version of this:

def while_equal(seq, other):
    for this, that in zip(seq, other):
        if this != that:
            return
        yield 1

def match(seq, other):
    return sum(while_equal(seq, other))

性能优于纯循环版本:

a = [0, 1, 2, 3, 4]
b = [0, 1, 2, 3, 4, 0]

import timeit

print(timeit.timeit('match_loop(a,b)', 'from __main__ import a, b, match_loop'))
print(timeit.timeit('match(a,b)', 'from __main__ import match, a, b'))

给予:

1.3171300539979711
1.291257290984504

也就是说，如果我们将纯循环版本改进为更加Pythonic:

That said, if we improve the pure loop version to be more Pythonic:

def match_loop(seq, other):
    count = 0
    for this, that in zip(seq, other):
        if this != that:
            return count
        count += 1
    return count

对我来说，这一次(使用与上面相同的方法)在0.8548871780512854处，比其他任何方法都快得多，同时仍然可读.这可能是由于原始版本中的按索引循环(通常非常慢)所致.但是，我认为本文是第一个版本，因为我认为它是最易读的.

This times (using the same method as above) at 0.8548871780512854 for me, significantly faster than any other method, while still being readable. This is probably due to looping by index in the original version, which is generally very slow. I, however, would go for the first version in this post, as I feel it's the most readable.

这篇关于库itertools与python代码相比的性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！