Python的使用多个处理器 [英] Python utilizing multiple processors

问题描述

可以说我有不同的长度需要转换或不同的需要进行调整或类似的东西大小的图像，音乐的大名单。顺序并不重要，因此非常适合跨多个处理器分裂。

Lets say I have a big list of music of varying length that needs to be converted or images of varying sizes that need to be resized or something like that. The order doesn't matter so it is perfect for splitting across multiple processors.

如果我用multiprocessing.Pool的地图功能，好像所有的工作被分割的时间提前，并没有考虑到的是，有些文件可能需要更长的时间做别人。

If I use multiprocessing.Pool's map function it seems like all the work is divided up ahead of time and doesn't take into account the fact that some files may take longer to do that others.

什么情况是，如果我有12个处理器...附近处理结束，1或2处理器将有2个或3个文件留下来处理而也可以使用其它的处理器处于闲置状态。

What happens is that if I have 12 processors... near the end of processing, 1 or 2 processors will have 2 or 3 files left to process while other processors that could be utilized sit idle.

是否有某种队列实现的，能保持加载所有的处理器，直到没有更多的工作来办？

Is there some sort of queue implementation that can keep all processors loaded until there is no more work left to do?

推荐答案

有一个的 队列 的多处理模块中类专门用于这一目的。

There is a Queue class within the multiprocessing module specifically for this purpose.

编辑：如果你正在寻找一个并行计算的一个完整的框架，它设有一个地图使用任务队列（）功能，看看的并行计算设施 IPython的。在particlar，您可以使用 TaskClient.map（） 函数来获得负载平衡的映射可用的处理器。

If you are looking for a complete framework for parallel computing which features a map() function using a task queue, have a look at the parallel computing facilities of IPython. In particlar, you can use the TaskClient.map() function to get a load-balanced mapping to the available processors.

这篇关于Python的使用多个处理器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！