python multiprocessing vs threading for Windows和Linux上的cpu绑定工作 [英] python multiprocessing vs threading for cpu bound work on windows and linux

问题描述

因此，我敲了一些测试代码，以了解与线程相比，多处理模块如何在cpu绑定工作上扩展.在linux上，我获得了预期的性能提升:

So I knocked up some test code to see how the multiprocessing module would scale on cpu bound work compared to threading. On linux I get the performance increase that I'd expect:

linux (dual quad core xeon):
serialrun took 1192.319 ms
parallelrun took 346.727 ms
threadedrun took 2108.172 ms

我的双核Macbook Pro表现出相同的行为:

My dual core macbook pro shows the same behavior:

osx (dual core macbook pro)
serialrun took 2026.995 ms
parallelrun took 1288.723 ms
threadedrun took 5314.822 ms

然后我去Windows机器上尝试了一下，结果却大不相同.

I then went and tried it on a windows machine and got some very different results.

windows (i7 920):
serialrun took 1043.000 ms
parallelrun took 3237.000 ms
threadedrun took 2343.000 ms

为什么哦，为什么在Windows上的多处理方法这么慢?

Why oh why, is the multiprocessing approach so much slower on windows?

这是测试代码:

#!/usr/bin/env python

import multiprocessing
import threading
import time

def print_timing(func):
    def wrapper(*arg):
        t1 = time.time()
        res = func(*arg)
        t2 = time.time()
        print '%s took %0.3f ms' % (func.func_name, (t2-t1)*1000.0)
        return res
    return wrapper


def counter():
    for i in xrange(1000000):
        pass

@print_timing
def serialrun(x):
    for i in xrange(x):
        counter()

@print_timing
def parallelrun(x):
    proclist = []
    for i in xrange(x):
        p = multiprocessing.Process(target=counter)
        proclist.append(p)
        p.start()

    for i in proclist:
        i.join()

@print_timing
def threadedrun(x):
    threadlist = []
    for i in xrange(x):
        t = threading.Thread(target=counter)
        threadlist.append(t)
        t.start()

    for i in threadlist:
        i.join()

def main():
    serialrun(50)
    parallelrun(50)
    threadedrun(50)

if __name__ == '__main__':
    main()

推荐答案

在UNIX变体下，过程更加轻巧. Windows进程很繁琐，需要花费更多的时间来启动.推荐使用线程在Windows上进行多处理.

Processes are much more lightweight under UNIX variants. Windows processes are heavy and take much more time to start up. Threads are the recommended way of doing multiprocessing on windows.

这篇关于python multiprocessing vs threading for Windows和Linux上的cpu绑定工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！