多处理池示例(并行)比顺序池慢.试图了解python中的池 [英] Multiprocessing pool example (parallel) is slower than sequential. Trying to understand pool in python

问题描述

我一直在阅读并尝试了解如何通过多处理更快地运行python.我找到了以下示例: multiprocessing.Pool示例.

I have been reading and trying to understand how to run python faster with multiprocessing. I found this example: multiprocessing.Pool example.

然后，我完成了此测试，并在具有8个内核的服务器中运行.

I then made up this test and ran in a server with 8 cores.

import time
from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    start_time = time.time()
    map(f, [x for x in range(1000000)])
    print("Sequential run time: %.2f seconds" % (time.time() - start_time))

    start_time = time.time()
    p = Pool(8)
    p.map(f, [x for x in range(1000000)])
    print("Parallel run time: %.2f seconds" % (time.time() - start_time))

但是，使用Pool

输出

Sequential run time: 0.13 seconds
Parallel run time: 0.98 seconds

我做错了什么? 谢谢

推荐答案

您没有做错任何事情，只是可能运行的进程超出了物理内核的运行时间-不能从这里猜测您是否拥有至少8个可用的内核来运行您创建的8个进程.

You're not doing anything wrong, except perhaps running more processes than you have physical cores to run them on - can't guess from here whether you have at least 8 cores available to run the 8 processes you created.

但是，即使这样做，与将参数传递到工作进程并将参数从工作进程传递回的所有开销相比，计算一个f(x)结果所花费的时间也微不足道. . IPC(进程间通信)并不便宜.

But even if you do, the time taken to compute one f(x) result is trivial compared to all the overheads of passing the arguments to, and passing the results back from, the worker processes - f() does very little work. IPC (inter-process communication) isn't cheap.

这就是为什么文档反复强调要做尽可能少的IPC的原因.如果每个函数调用完成的有用工作很少，那么您将不会获得整体的提速(相反，您已经发现).

That's why the docs stress repeatedly to do as little IPC as possible. You're not going to get any overall speedup (to the contrary, as you've found) if the useful work done per function call is tiny.

顺便说一句，这个:

[x for x in range(1000000)]

最好写成普通的:

range(1000000)

尽管进行此更改与此处的实际要点没有区别.

although making that change makes no difference to the real points here.

这篇关于多处理池示例(并行)比顺序池慢.试图了解python中的池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！