如何在Python中使用多重处理并行求和循环 [英] How to parallel sum a loop using multiprocessing in Python

问题描述

我很难理解如何使用Python的多处理模块.

I am having difficulty understanding how to use Python's multiprocessing module.

我有一个从1到n的总和，其中n=10^10太大了，无法放入列表中，这似乎是许多使用多重处理的在线示例的推力.

I have a sum from 1 to n where n=10^10, which is too large to fit into a list, which seems to be the thrust of many examples online using multiprocessing.

是否可以将范围拆分"为一定大小的分段，然后对每个分段进行求和?

Is there a way to "split up" the range into segments of a certain size and then perform the sum for each segment?

例如

def sum_nums(low,high):
    result = 0
    for i in range(low,high+1):
        result += i
    return result

我想通过将sum_nums(1,10**10)分解为许多sum_nums(1,1000) + sum_nums(1001,2000) + sum_nums(2001,3000)...等来进行计算.我知道有一个简短的n(n+1)/2形式，但假装我们不知道.

And I want to compute sum_nums(1,10**10) by breaking it up into many sum_nums(1,1000) + sum_nums(1001,2000) + sum_nums(2001,3000)... and so on. I know there is a close-form n(n+1)/2 but pretend we don't know that.

这是我尝试过的

import multiprocessing

def sum_nums(low,high):
    result = 0
    for i in range(low,high+1):
        result += i
    return result

if __name__ == "__main__":
    n = 1000 
    procs = 2 

    sizeSegment = n/procs

    jobs = []
    for i in range(0, procs):
        process = multiprocessing.Process(target=sum_nums, args=(i*sizeSegment+1, (i+1)*sizeSegment))
        jobs.append(process)

    for j in jobs:
        j.start()
    for j in jobs:
        j.join()

    #where is the result?

推荐答案

首先，解决内存问题的最佳方法是使用迭代器/生成器而不是列表:

First, the best way to get around the memory issue is to use an iterator/generator instead of a list:

def sum_nums(low, high):
    result = 0
    for i in xrange(low, high+1):
        result += 1
    return result

在python3中，range()生成一个迭代器，因此仅在python2中需要

现在，要想将处理拆分到不同的进程或CPU内核时，就可以进行多处理.如果您不需要控制单个工人，那么最简单的方法就是使用流程池.这将使您将函数映射到池并获取输出.您也可以使用apply_async一次将作业应用到一个池中，并获得延迟的结果，该结果可以通过.get()获得:

Now, where multiprocessing comes in is when you want to split up the processing to different processes or CPU cores. If you don't need to control the individual workers than the easiest method is to use a process pool. This will let you map a function to the pool and get the output. You can alternatively use apply_async to apply jobs to the pool one at a time and get a delayed result which you can get with .get():

import multiprocessing
from multiprocessing import Pool
from time import time

def sum_nums(low, high):
    result = 0
    for i in xrange(low, high+1):
        result += i
    return result

# map requires a function to handle a single argument
def sn((low,high)):
    return sum_nums(low, high) 

if __name__ == '__main__': 
    #t = time()
    # takes forever   
    #print sum_nums(1,10**10)
    #print '{} s'.format(time() -t)
    p = Pool(4)

    n = int(1e8)
    r = range(0,10**10+1,n)
    results = []

    # using apply_async
    t = time()
    for arg in zip([x+1 for x in r],r[1:]):
        results.append(p.apply_async(sum_nums, arg))

    # wait for results
    print sum(res.get() for res in results)
    print '{} s'.format(time() -t)

    # using process pool
    t = time()
    print sum(p.map(sn, zip([x+1 for x in r], r[1:])))
    print '{} s'.format(time() -t)

在我的计算机上，仅用10 ** 10调用sum_nums大约需要9分钟，但是使用Pool(8)和n=int(1e8)会将其减少到一分钟以上.

On my machine, just calling sum_nums with 10**10 takes almost 9 minutes, but using a Pool(8) and n=int(1e8) reduces this to just over a minute.

这篇关于如何在Python中使用多重处理并行求和循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！