良好的多处理示例实现? [英] Good example implementation of multiprocessing?

问题描述

我正在尝试将我的程序之一转换为使用多处理程序，最好是使用多处理程序池，因为它们看起来更简单.从较高的角度来看，该过程是根据图像创建一系列补丁，然后将它们传递给GPU进行对象检测. CPU和GPU部分各花费大约4s，但是CPU有8个内核，并且不必等待GPU，因为在数据通过GPU之后，无需对数据进行任何进一步的操作.

I am trying to convert one of my programs to use multiprocessing, preferably the multiprocessing pools since those seem simpler to do. At a high level the process is creating an array of patches from images and then passing them to the GPU for object detection. The CPU and GPU part take about 4s each, however the CPU has 8 cores and it doesn't have to wait for the GPU because no further operations are done to the data after it passes the GPU.

以下是我想象这应该如何工作的图表:

Here is a diagram of how I imagine this should work:

为帮助完成该过程，我希望对实现的高层版本进行演示.假设我们正在循环浏览包含10张图片的文件夹中的图片列表.我们一次调整图像4的大小.然后我们一次将它们转换为黑白两种，我们可以将转换作为此处过程的GPU部分.代码如下所示:

To help the process along I would like a demonstration with a high level version of my implementation. Say we are looping through a list of images in a folder that has 10 images. We resize images 4 at a time. Then we convert them to black and white two at a time, we can take the conversion as the GPU part of the process here. Here is what the code would look like:

def im_resize(im, num1, num2):
    return im.resize((num1, num2), Image.ANTIALIAS)

def convert_bw(im):
    return im.convert('L')

def read_images(path):
    imlist = []
    for pathAndFileName in glob.iglob(os.path.join(path, "*")):
        if pathAndFileName.endswith(tuple([".jpg", ".JPG"])):
            imlist.append(Image.open(pathAndFileName))
    return imlist


img_list = read_images("path/to/images/")
final_img_list = []

for image in img_list:

    # Resize needs to run concurrently on 4 processes so that the next img_tmp is always ready to go for convert
    img_tmp = im_resize(image, 100, 100)

    # Convert is limited, need to run on 2 processes
    img_tmp = convert_bw(img_tmp)
    final_img_list.append(img_tmp)

特定数量进程的原因是由于系统性能指标所致，这将减少运行时间.我只是想确保GPU不必等待CPU完成图像处理，并且我想让常量队列中充满预处理图像，以供GPU运行.我最好希望在队列中保留大约4-10张经过预处理的图像的最大大小.如果你们可以通过简化的示例帮助我说明如何实现这一目标，那么我相信我可以弄清楚如何将其转化为我的需求.

The reason for the specific number of processes and such is due to system performance metrics, this is what will reduce the runtime. I just want to make sure that the GPU doesn't have to be waiting for the CPU to finish processing images, and I want to have a constant queue filled with pre-processed images ready for the GPU to run. I would preferably want to keep a maximum size on the queue of about 4-10 pre-processed images. If you guys can help me illustrate how I would achieve this with this simplified example I'm sure I can figure out how to translate it into what I need for mine.

谢谢！

推荐答案

以下是尝试实现所需内容的尝试:

Here's a tentative attempt at implementing what you want:

...

# Mapping functions can only take one arg, we provide tuple
def img_resize_splat(a):
    img_resize(*a)

if __name__=="__main__":
    # Make a CPU pool and a GPU pool
    cpu = Pool(4)
    gpu = Pool(2)

    # Hopefully this returns an iterable, and not a list with all images read into memory
    img_list = read_images("path/to/images/")

    # I'm assuming you want images to be processed as soon as ready, order doesn't matter
    resized = cpu.imap_unordered(img_resize_splat, ((img, 100, 100) for img in img_list))
    converted = gpu.imap_unordered(convert_bw, resized)

    # This is an iterable with your results, slurp them up one at a time
    for bw_img in converted:
        # do something

这篇关于良好的多处理示例实现?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！