列表的并行处理 [英] Parallel processing of lists

问题描述

我正在努力使自己处理多处理.我有一个列表，将其分为两个相等的长部分，并在两个单独的过程中对它们进行排序.我知道这部分有效，因为打印saveto给了我两个列表.但是我无法访问它们，因为最后我得到了两个空列表.为什么我无法访问要写入l1和l2的内容，我该怎么做?

I'm trying to get my head around multiprocessing. I have a list, I divide it in two equally long parts, I sort them in two separate processes. I know that this part works because printing saveto gives me the two lists. But I can't access them because in the end I get two empty lists. Why can't I access what I want to be written to l1 and l2 and how do I do that?

import multiprocessing
import random


def sort(l, saveto):
    saveto = sorted(l)
    print saveto

if __name__ == '__main__':

    l = [int(100000*random.random()) for i in xrange(10000)]

    listlen = len(l)
    halflist = listlen/2
    l1 = []
    l2 = []

    p1 = multiprocessing.Process(target=sort, args=(l[0:halflist], l1))
    p2 = multiprocessing.Process(target=sort, args=(l[halflist:listlen], l2))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    print l1
    print l2

推荐答案

使用 multiprocessing.Queue 在进程之间共享数据

Use multiprocessing.Queue to share data between processes

import multiprocessing
import random

def sort(l, queue):
    queue.put(sorted(l))


if __name__ == '__main__':
    l = [int(100000*random.random()) for i in xrange(10000)]

    listlen = len(l)
    halflist = listlen/2
    queue = multiprocessing.Queue()

    p1 = multiprocessing.Process(target=sort, args=(l[0:halflist], queue))
    p2 = multiprocessing.Process(target=sort, args=(l[halflist:listlen], queue))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    print queue.get()
    print queue.get()

更新:

事实证明，将大量数据放入Queue可能会导致死锁.这是文档中提到的 :

As it turned out putting the large amounts of data to Queue can cause a deadlock. This is mentioned in the docs:

警告

如上所述，如果子进程已将项目放入队列中(并且 它没有使用JoinableQueue.cancel_join_thread)，那么该过程 直到所有缓冲的项目都已刷新到 管道.

As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.

这意味着，如果您尝试加入该过程，则可能会陷入僵局. 除非您确定所有已放入队列中的项目 已被消耗.同样，如果子进程是非守护进程 那么当父进程尝试加入其所有进程时，其父进程可能会在退出时挂起 非守护儿童.

This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.

请注意，使用管理器创建的队列不存在此问题.

Note that a queue created using a manager does not have this issue.

固定版本:

import multiprocessing
import random

def sort(l, queue):
    queue.put(sorted(l))


if __name__ == '__main__':
    l = [int(100000*random.random()) for i in range(10000)]

    listlen = len(l)
    halflist = listlen/2

    manager = multiprocessing.Manager()
    queue = manager.Queue()

    p1 = multiprocessing.Process(target=sort, args=(l[0:halflist], queue))
    p2 = multiprocessing.Process(target=sort, args=(l[halflist:listlen], queue))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    print queue.get()
    print queue.get()

这篇关于列表的并行处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！