在Python中进行多处理要比没有处理慢 [英] Multiprocessing in Python slower than without

问题描述

我在具有2个核心但4个线程的计算机上使用python 3.3.我正在尝试学习使用多处理来加速代码，但是使用它时，我的代码会变慢.

为了开始学习，我制作了一个小程序:

from multiprocessing import Process
import time

def f():
    s = 0
    for i in range(2*10**7):
        s += i
    return s

if __name__ == '__main__':
    t = time.time()
    p1 = Process(target = f)
    p2 = Process(target = f)
    p3 = Process(target = f)
    p4 = Process(target = f)
    p1.start()
    p2.start()
    p3.start()
    p4.start()
    p1.join()
    p2.join()
    p3.join()
    p4.join()
    print (time.time()-t)

t2 = time.time()
for a in range(4):
    f()
print(time.time()-t2)

平均运行3次，带有多处理的第一部分需要17.15秒，而没有多处理的第二部分需要6.24秒.使用Windows任务管理器，我发现计算机的第一部分确实使用了100％的CPU，第二部分仅使用了25％的CPU，而且我的内存还没有耗尽.

为什么该程序的多处理速度这么慢?

解决方案

Windows没有fork()，因此每次启动新进程时，多处理必须通过导入__main__模块来解决.

这意味着当每个子进程运行时，它不仅运行目标函数，而且还运行文件末尾的部分.将其移入块中，应该会更快！

I'm using python 3.3 on a computer with 2 cores but 4 threads. I am trying to learn to use multiprocessing to speed up code, but when using it my code slows down.

To start my learning, I have made a small program:

from multiprocessing import Process
import time

def f():
    s = 0
    for i in range(2*10**7):
        s += i
    return s

if __name__ == '__main__':
    t = time.time()
    p1 = Process(target = f)
    p2 = Process(target = f)
    p3 = Process(target = f)
    p4 = Process(target = f)
    p1.start()
    p2.start()
    p3.start()
    p4.start()
    p1.join()
    p2.join()
    p3.join()
    p4.join()
    print (time.time()-t)

t2 = time.time()
for a in range(4):
    f()
print(time.time()-t2)

Average of 3 runs, the first part with multiprocessing takes 17.15 sec, while the second part without multiprocessing takes 6.24 sec. Using the Windows Task Manager, I see that my computer is indeed using 100% CPU for the first part and only 25% for the second part, and also that I am not running out of memory.

Why is this program so much slower with multiprocessing?

解决方案

Windows has no fork(), so multiprocessing has to work around that by importing the __main__ module each time a new process is started.

This means when each of your subprocesses runs, it doesn't only run the target function, but also the part at the end of the file. Move it into the block and it should be way faster!

这篇关于在Python中进行多处理要比没有处理慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！