itertools.islice与列表切片进行比较 [英] itertools.islice compared to list slice

问题描述

我一直在尝试应用算法，根据特定条件将python列表缩小为较小的列表。由于原始列表的大量，大约100k元素，我尝试使用itertools来避免多个内存分配，所以我提出了这个：

I've been trying to apply an algorithm to reduce a python list into a smaller one based on a certain criteria. Due to the large volume of the original list, in the order of 100k elements, I tried to itertools for avoiding multiple memory allocations so I came up with this:

reducedVec = [ 'F' if sum( 1 for x in islice(vec, i, i+ratio) if x == 'F' ) 
                         > ratio / 3.0 else 'T'
                for i in xrange(0, len(vec), ratio) ]

当vec有大约100k元素时，执行时间大约需要几分钟。当我尝试改为：

Execution time for this takes a worryingly long time in the order of a few minutes, when vec has around 100k elements. When I tried instead:

reducedVec = [ 'F' if sum( 1 for x in vec[i:i+ratio] if x == 'F' ) 
                         > ratio / 3.0 else 'T'
                for i in xrange(0, len(vec), ratio) ]

本质上用切片替换islice执行是瞬时的。

in essence replace islice with a slice the execution is instantaneous.

你能想到一个合理的解释吗？我本以为避免重复分配一个包含大量元素的新列表，实际上可以节省一些计算周期而不是削弱整个执行。

Can you think of a plausible explanation for this? I would have thought that avoiding to repeatedly allocate a new list with a substantial number of elements, would actually save me a few computational cycles instead of crippling the whole execution.

干杯，
Themis

Cheers, Themis

推荐答案

islice 适用于任意迭代。要做到这一点，不是直接跳到第n个元素，而是必须迭代第一个n-1，抛弃它们，然后产生你想要的那个。

islice works with arbitrary iterables. To do this, rather than jumping straight to the nth element, it has to iterate over the first n-1, throwing them away, then yield the ones you want.

查看 itertools文档中的纯Python实现：

Check out the pure Python implementation from the itertools documentation:

def islice(iterable, *args):
    # islice('ABCDEFG', 2) --> A B
    # islice('ABCDEFG', 2, 4) --> C D
    # islice('ABCDEFG', 2, None) --> C D E F G
    # islice('ABCDEFG', 0, None, 2) --> A C E G
    s = slice(*args)
    it = iter(xrange(s.start or 0, s.stop or sys.maxint, s.step or 1))
    nexti = next(it)
    for i, element in enumerate(iterable):
        if i == nexti:
            yield element
            nexti = next(it)

---

说到itertools文档，如果我尝试执行此操作，我会可能使用石斑鱼配方。它实际上不会为你节省任何记忆，但如果你把它重写为更懒惰，这可能不会很难。

---

Speaking of the itertools documentation, if I was trying to do this operation, I'd probably use the grouper recipe. It won't actually save you any memory, but it could if you rewrote it to be lazier, which wouldn't be tough.

from __future__ import division

from itertools import izip_longest
def grouper(n, iterable, fillvalue=None):
    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

reducedVec = []
for chunk in grouper(ratio, vec):
    if sum(1 for x in chunk if x == 'F') > ratio / 3:
        reducedVec.append('F')
    else:
        reducedVec.append('T')

我喜欢使用 grouper 来抽象出连续的切片，并且发现这段代码比原来的更容易阅读

I like using grouper to abstract away the consecutive slices and find this code a lot easier to read than the original

这篇关于itertools.islice与列表切片进行比较的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！