python generator:并行解压整个发电机 [英] python generator: unpack entire generator in parallel

问题描述

假设我有一个生成器，其__next__()函数有些昂贵，并且我想尝试并行化调用.我应该在哪里进行并行化处理?

Suppose I have a generator whose __next__() function is somewhat expensive and I want to try to parallelize the calls. Where do I throw in the parallization?

要更具体一点，请考虑以下示例:

To be slightly more concrete, consider this example:

# fast, splitting a file for example
raw_blocks = (b for b in block_generator(fin))
# slow, reading blocks, checking values ...
parsed_blocks = (block_parser(b) for b in raw_blocks)
# get all parsed blocks into a data structure
data = parsedBlocksToOrderedDict(parsed_blocks)

最基本的是将第二行更改为进行并行化的内容.是否存在一些生成器魔咒，可以让人们并行打开(第3条)生成器包的包装?并行调用__next__()?

The most basic thing is to change the 2nd line to something that does the parallelization. Is there some generator magic that allows one to unpack the generator (on the 3rd) line in parallel? Calling __next__() in parallel?

推荐答案

否.您必须按顺序调用next()，因为任何非平凡生成器的下一个状态均由其当前状态决定.

No. You must call next() sequentially because any non-trivial generator's next state is determined by its current state.

def gen(num):
    j=0
    for i in xrange(num):
        j += i
        yield j

在不知道生成每一个值的状态下，无法并行化对上述生成器的调用.但是，如果您知道这一点，则无需运行它.

There's no way to parallelize calls to the above generator without knowing its state at each point it yields a value. But if you knew that, you wouldn't need to run it.

这篇关于python generator:并行解压整个发电机的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！