适用于并行进程的Python多处理 [英] Python multiprocessing for parallel processes
问题描述
对于某些人来说这太简单了，我感到抱歉，但是我仍然无法理解python的多处理技巧.我读过
http://docs.python.org/dev/library/multiprocessing
http://pymotw.com/2/multiprocessing/basics.html
以及Google提供给我的许多其他教程和示例...很多也是从这里开始的.
I'm sorry if this is too simple for some people, but I still don't get the trick with python's multiprocessing. I've read
http://docs.python.org/dev/library/multiprocessing
http://pymotw.com/2/multiprocessing/basics.html
and many other tutorials and examples that google gives me... many of them from here too.
好吧，我的情况是我必须计算许多numpy矩阵，然后需要将它们存储在单个numpy矩阵中.假设我要使用20个核心(或者我可以使用20个核心)，但是由于它一直使进程保持活动状态直到池消失"，因此我没有成功使用池资源.所以我想做这样的事情:
Well, my situation is that I have to compute many numpy matrices and I need to store them in a single numpy matrix afterwards. Let's say I want to use 20 cores (or that I can use 20 cores) but I haven't managed to successfully use the pool resource since it keeps the processes alive till the pool "dies". So I thought on doing something like this:
from multiprocessing import Process, Queue
import numpy as np
def f(q,i):
q.put( np.zeros( (4,4) ) )
if __name__ == '__main__':
q = Queue()
for i in range(30):
p = Process(target=f, args=(q,))
p.start()
p.join()
result = q.get()
while q.empty() == False:
result += q.get()
print result
，但是看起来这些进程不是并行运行，而是顺序运行(如果我错了，请纠正我)，而且我不知道它们在执行计算后是否会死亡(因此超过20个)处理那些将自己的职责留给核心的资源，以供其他流程使用).另外，对于一个非常大的数字(比如说100.000)，将所有这些矩阵(可能也真的很大)存储在队列中将占用大量内存，使代码无用，因为这种想法是将每个结果都放在每次迭代中在最终结果中，就像使用锁(及其锁(及其Acquisition()和release()方法)一样)，但是如果此代码不用于并行处理，则锁也无济于事...
but then it looks like the processes don't run in parallel but they run sequentially (please correct me if I'm wrong) and I don't know if they die after they do their computation (so for more than 20 processes the ones that did their part leave the core free for another process). Plus, for a very large number (let's say 100.000), storing all those matrices (which may be really big too) in a queue will use a lot of memory, rendering the code useless since the idea is to put every result on each iteration in the final result, like using a lock (and its acquire() and release() methods), but if this code isn't for parallel processing, the lock is useless too...
我希望有人可以帮助我.
I hope somebody may help me.
提前谢谢！
推荐答案
您是正确的，它们在您的示例中按顺序执行.
You are correct, they are executing sequentially in your example.
p.join()导致当前线程阻塞，直到完成执行为止.您可能想在for循环之外单独加入您的进程(例如，通过将它们存储在列表中，然后对其进行迭代)，或者将numpy.Pool和apply_async之类的东西与回调一起使用.这样一来，您也可以直接将其添加到结果中，而不必保留对象.
p.join() causes the current thread to block until it is finished executing. You'll either want to join your processes individually outside of your for loop (e.g., by storing them in a list and then iterating over it) or use something like numpy.Pool and apply_async with a callback. That will also let you add it to your results directly rather than keeping the objects around.
例如:
def f(i):
return i*np.identity(4)
if __name__ == '__main__':
p=Pool(5)
result = np.zeros((4,4))
def adder(value):
global result
result += value
for i in range(30):
p.apply_async(f, args=(i,), callback=adder)
p.close()
p.join()
print result
最后关闭然后再加入池可确保池的进程已完成，并且result对象已完成计算.您也可以使用Pool.imap作为问题的解决方案进行调查.该特定解决方案看起来像这样:
Closing and then joining the pool at the end ensures that the pool's processes have completed and the result object is finished being computed. You could also investigate using Pool.imap as a solution to your problem. That particular solution would look something like this:
if __name__ == '__main__':
p=Pool(5)
result = np.zeros((4,4))
im = p.imap_unordered(f, range(30), chunksize=5)
for x in im:
result += x
print result
这对于您的特定情况比较干净，但对于最终尝试执行的操作而言可能并非如此.
This is cleaner for your specific situation, but may not be for whatever you are ultimately trying to do.
关于存储所有变化的结果，如果我理解您的问题，您可以使用imap/
As to storing all of your varied results, if I understand your question, you can just add it off into a result in the callback method (as above) or item-at-a-time using imap/imap_unordered (which still stores the results, but you'll clear it as it builds). Then it doesn't need to be stored for longer than it takes to add to the result.
这篇关于适用于并行进程的Python多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
