为什么多处理很慢 [英] Why multiprocessing is slow

问题描述

为了加速我的程序，我刚刚开始阅读有关多处理的内容.因此，我编写了两个基本示例来从随机数列表中提取素数.

I just started reading about multiprocessing for the sake of speeding up my programs. hence, i wrote two basic examples to extract prime numbers from a list of random numbers.

示例 1:使用多处理

from multiprocessing import Process, Queue
from random import randrange
import time

def randomList(q, size, nmax):
    l = []
    r = randrange(2, nmax)
    for i in range(size):
        while r in l: # avoid replicating numbers
            r = randrange(2, nmax)
        l.append(r)
        q.put(r)

def checkPrime(numbers, prime):
    if numbers.qsize():
        n = numbers.get()
        count = 0 # divisors counter
        d = 2 # divisor
        while not count and d<=n/2:
            if n%d:
                d+=1
            else:
                count+=1
        if not count:
            prime.put(n)

if __name__=="__main__":
    numbers = Queue()
    prime = Queue()
    randomList(numbers, 50, 1000) # 50 number | 100 max value
    t1 = time.time()
    while numbers.qsize():
        for i in range(10): # Running 10 processes 
            p=Process(target=checkPrime, args=(numbers, prime))
            p.start()
            p.join()
    t2 = time.time()
    primes = []
    for i in range(prime.qsize()):
        primes.append(prime.get())
    print("[+] Prime numbers:")
    print(primes)
    print("[+] Time elapsed:"+str(t2-t1))

输出:

[+] Prime numbers:
[17, 227, 389, 593, 953, 757]
[+] Time elapsed:9.41699981689

示例 2:与示例 1 相同但没有多处理

Exemple 2: the same example 1 but without multiprocessing

[...]
    while numbers.qsize():
        checkPrime(numbers, prime)
[...]

输出:

[+] Prime numbers:
[193, 227, 241, 439, 499, 877, 479, 743, 929]
[+] Time elapsed:0.00999999046326

因此，多处理使这个程序(特别是可能)比不使用它慢得多.有什么解释吗?我用错了吗?

So, multiprocessing makes this program (specifically maybe) hugely slower than without using it. Any explanation? am I using it the wrong way?

推荐答案

我认为您的多处理方法很差.与其将工作分成 10 个进程并同时启动它们，不如一次启动一个进程，每个进程都在做一个工作单元，然后退出.您的实现将在其生命周期内创建(然后销毁)50 个进程，这会产生大量开销.

I'm thinking your method of multiprocessing is poor. Rather than splitting the work into 10 processes and starting them all up at once, you're starting a single process at a time and each one is doing a single unit of work then exiting. Your implementation will create (and then destroy) 50 processes over its lifetime, which creates a lot of overhead.

您还可以在启动进程后立即加入这些进程，这样您就不会真正运行多个进程.加入使得它等待子进程完成后再继续.

You're also joining the processes immediately after starting them, which will make it so you're never actually running multiple processes. The join makes it wait for the child process to finish before continuing.

最后，必须有一种更好的方法来返回结果，即使用队列并一次获取一个值.如果你可以一次启动每个进程并做一组工作，然后将列表中的结果返回给主线程，你可以减少使用队列的开销.

Finally, there's gotta be a better way to return results that using a queue and getting a single value at a time. If you can start each process at once with a set of work to do and then return the results in list to the master thread, you could reduce the overhead of using the Queue.

这篇关于为什么多处理很慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！