python中跨多个进程的同步 [英] synchronization across multiple processes in python

问题描述

我有一个 python 应用程序，它产生一个单独的进程来做一些工作(由于 GIL(全局解释器锁)，我在使用线程时遇到了性能问题).现在我在python中同步跨进程共享资源的方法是什么?

I have a python application that spawns a separate process to do some work (I ran into performance issues using threads due to the GIL (global interpreter lock)). Now what's my methods in python to synchronize shared resources across processes?

我将数据移动到一个队列中，然后一个 spawn 进程在从该队列接收数据时完成这项工作.但是我需要能够保证数据以有序的方式输出，与复制的顺序相同，因此我需要保证在任何时候只有一个进程可以从/向队列读/写.我该如何做到最好?

I move data into a queue and a spawn process does the work when as it receives data from that queue. But I need to be able to guarantee that the data is coming out in an orderly fashion, same order as it was copied in so I need to guarantee that only one process at any time can read/write from/to the queue. How do I do that best?

谢谢，罗恩

推荐答案

我觉得你需要一个 Semaphore，查看这个示例代码:

I think you need a Semaphore, check this example code:

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        pool.acquire()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)
        pool.release()


pool = threading.BoundedSemaphore(value=1)


for i in range(10):
        t = ThreadClass()
        t.start()

有这个输出:

Thread-1 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-2 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-3 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-4 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-5 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-6 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-7 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-8 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-9 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-10 says hello, World! at time: 2013-05-20 18:57:47.609000

如:

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)




for i in range(10):
        t = ThreadClass()
        t.start()

有这个输出:

Thread-1 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-2 says hello, World! at time: 2013-05-20 18:58:05.
531000

 Thread-4 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-3 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-6 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-5 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-8 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-7 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-10 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-9 says hello, World! at time: 2013-05-20 18:58:0
5.531000

因此，使用 python 中的 BoundedSemaphore，您可以确保在任何人写入您的队列之前，他们必须拥有信号量.但这并不能确保您的结果以正确的顺序添加到队列中.

So with a BoundedSemaphore in python you can make sure that before anyone writes to your queue they have to have the semaphore. This doesn't ensure your results are added to the queue in the correct order though.

如果您要这样做并保持订单，您将需要这样的东西:

If you going to do this and keep order your going to need something like this:

import multiprocessing
import datetime
import random
import time

def funfun(number):
    time.sleep(random.randint(0,10))
    now = datetime.datetime.now()
    return "%s says hello, World! at time: %s"  % (number,now)

if __name__ == "__main__":
    pool = multiprocessing.Pool(10)
    for item in pool.imap(funfun,[i for i in range(10)]):
        print item

打印:

0 says hello, World! at time: 2013-05-21 00:38:48.546000
1 says hello, World! at time: 2013-05-21 00:38:55.562000
2 says hello, World! at time: 2013-05-21 00:38:47.562000
3 says hello, World! at time: 2013-05-21 00:38:51.578000
4 says hello, World! at time: 2013-05-21 00:38:50.578000
5 says hello, World! at time: 2013-05-21 00:38:48.593000
6 says hello, World! at time: 2013-05-21 00:38:52.593000
7 says hello, World! at time: 2013-05-21 00:38:48.593000
8 says hello, World! at time: 2013-05-21 00:38:50.593000
9 says hello, World! at time: 2013-05-21 00:38:51.609000

因此，您可以按照正确的顺序添加到队列中，作业将等待轮到它们添加到队列中.

So with this you could just append to the queue in the right order, and the jobs will wait for their turn to add to the queue.

这篇关于python中跨多个进程的同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！