具有两个参数的多进程itertool组合 [英] Multiprocess itertool combination with two arguments

问题描述

我想使用多处理程序运行以下功能:

I have the following function that I would like to run using multiprocessing:

def bruteForcePaths3(paths, availableNodes):

    results = []

    #start by taking each combination 2 at a time, then 3, etc
    for i in range(1,len(availableNodes)+1):
        print "combo number: %d" % i

        currentCombos = combinations(availableNodes, i)

        for combo in currentCombos:
            #get a fresh copy of paths for this combiniation
            currentPaths = list(paths)
            currentRemainingPaths = []
            # print combo

            for node in combo:
                #determine better way to remove nodes, for now- if it's in, we remove
                currentRemainingPaths = [path for path in currentPaths if not (node in path)]
                currentPaths = currentRemainingPaths

            #if there are no paths left
            if len(currentRemainingPaths) == 0:
                #save this combination
                print combo
                results.append(frozenset(combo))

    return results 

基于其他几篇文章(将itertools和多处理结合起来?)，我试图通过以下步骤对此进行多处理:

Bases on a few other post (Combining itertools and multiprocessing?), I tried to multiprocess this by the following:

def grouper_nofill(n, iterable):
        it=iter(iterable)
        def take():
            while 1: yield list(islice(it,n))
        return iter(take().next,[])

    def mp_bruteForcePaths(paths, availableNodes):

        pool = multiprocessing.Pool(4)
        chunksize=256
        async_results=[]


        def worker(paths,combos, out_q):
            """ The worker function, invoked in a process. 'nums' is a
                list of numbers to factor. The results are placed in
                a dictionary that's pushed to a queue.
            """
            results = bruteForcePaths2(paths, combos)
            print results
            out_q.put(results)


        for i in range(1,len(availableNodes)+1):
            currentCombos = combinations(availableNodes, i)
            for finput in grouper_nofill(chunksize,currentCombos):

                args = (paths, finput)
                async_results.extend(pool.map_async(bruteForcePaths2, args).get())

                print async_results

    def bruteForcePaths2(args):
        paths, combos = args
        results = []

        for combo in combos:
            #get a fresh copy of paths for this combiniation
            currentPaths = list(paths)
            currentRemainingPaths = []
            # print combo

            for node in combo:
                #determine better way to remove nodes, for now- if it's in, we remove
                currentRemainingPaths = [path for path in currentPaths if not (combo in path)]
                currentPaths = currentRemainingPaths

            #if there are no paths left
            if len(currentRemainingPaths) == 0:
                #save this combination
                print combo
                results.append(frozenset(combo))

        return results

我需要能够将2个参数传递给bruteforce函数.我收到错误消息: 太多价值无法解包"

I need to be able to pass in 2 arguments to the bruteforce function. I'm getting the error: "too many values to unpack"

第3部分问题: 如何在nproc cpu拆分组合迭代器上对bruteforce函数进行多处理? 如何传递两个参数-路径和组合? 我如何获得结果(认为mpa_async应该为我做到这一点)?

So 3 part question: How can I multiprocess the bruteforce function over nproc cpu's splitting the combinations iterator? How can I pass in the two arguments- path and combinations? How do I get the result (think the mpa_async should do that for me)?

谢谢.

推荐答案

此

args = (paths, finput)
pool.map_async(bruteForcePaths2, args)

拨打这两个电话，这不是您的意图:

makes these two calls, which is not your intent:

bruteForcePaths2(paths)
bruteForcePaths2(finput)

您可以使用apply_async代替，以向池中提交单个函数调用.还要注意，如果立即调用get，它将等待结果，并且不会从多处理中获得任何好处.

You can use apply_async instead to submit single function calls to the pool. Note also that if you call get immediately it will wait for the result, and you don't get any advantage from multiprocessing.

您可以这样做:

for i in range(1,len(availableNodes)+1):
    currentCombos = combinations(availableNodes, i)
    for finput in grouper_nofill(chunksize,currentCombos):

        args = (paths, finput)
        async_results.append(pool.apply_async(bruteForcePaths2, args))

results = [x.get() for x in async_results]

这篇关于具有两个参数的多进程itertool组合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！