python 生成器:并行解包整个生成器 [英] python generator: unpack entire generator in parallel

问题描述

假设我有一个生成器，它的 __next__() 函数有点昂贵，我想尝试并行化调用.我在哪里投入平行化?

Suppose I have a generator whose __next__() function is somewhat expensive and I want to try to parallelize the calls. Where do I throw in the parallization?

更具体一点，请考虑以下示例:

To be slightly more concrete, consider this example:

# fast, splitting a file for example
raw_blocks = (b for b in block_generator(fin))
# slow, reading blocks, checking values ...
parsed_blocks = (block_parser(b) for b in raw_blocks)
# get all parsed blocks into a data structure
data = parsedBlocksToOrderedDict(parsed_blocks)

最基本的事情是将第二行更改为进行并行化的内容.是否有一些生成器魔法可以让一个人并行解压生成器(在第三行)?并行调用__next__()?

The most basic thing is to change the 2nd line to something that does the parallelization. Is there some generator magic that allows one to unpack the generator (on the 3rd) line in parallel? Calling __next__() in parallel?

推荐答案

没有.您必须按顺序调用 next()，因为任何非平凡生成器的下一个状态都由其当前状态决定.

No. You must call next() sequentially because any non-trivial generator's next state is determined by its current state.

def gen(num):
    j=0
    for i in xrange(num):
        j += i
        yield j

如果不知道生成值的每个点的状态，就无法并行化对上述生成器的调用.但如果你知道这一点，你就不需要运行它了.

There's no way to parallelize calls to the above generator without knowing its state at each point it yields a value. But if you knew that, you wouldn't need to run it.

这篇关于python 生成器:并行解包整个生成器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！