多处理嵌套 python 循环 [英] Multiprocessing nested python loops

问题描述

为了改进我的有一个重循环的代码，我需要加快速度.如何为这样的代码实现多处理?(a 是典型的大小 2 和 l 到 10)

To improve my code which has one heavy loop I need a speed up. How can I implement multiprocessing for a code like this? (a is typical of size 2 and l up to 10)

for x1 in range(a**l):
    for x2 in range(a**l):
        for x3 in range(a**l):
            output[x1,x2,x3] = HeavyComputationThatIsThreadSafe1(x1,x2,x3)

推荐答案

如果 HeavyComputationThatIsThreadSafe1 函数只使用数组而不使用 python 对象，我会使用 并发期货(或 python2 backport) ThreadPoolExecutor 以及 Numba (或 cython) 发布 GIL.否则使用 ProcessPoolExecutor.

If the HeavyComputationThatIsThreadSafe1 function only uses arrays and not python objects, I would using a concurrent futures (or the python2 backport) ThreadPoolExecutor along with Numba (or cython) with the GIL released. Otherwise use a ProcessPoolExecutor.

见:

http://numba.pydata.org/numba-doc/latest/user/examples.html#multi-threading

您希望在最外层循环的级别并行化计算，然后从每个线程/进程产生的块中填充 output.这假设这样做的成本比计算便宜得多，应该是这种情况.

You'd want to parallelize the calculation at the level of the outermost loop and and then fill output from the chunks resulting from each thread/process. This assumes the cost of doing so is much cheaper than the computation, which should be the case.

这篇关于多处理嵌套 python 循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！