如何使用Python和OpenCV进行多处理？ [英] How to use Python and OpenCV with multiprocessing?

问题描述

我正在使用Python 3.4.3和OpenCV 3.0.0来处理（应用各种过滤器）内存中非常大的图像（80,000 x 60,000），并且我想使用多个CPU内核来提高性能。经过一些阅读后，我得出了两种可能的方法：1）使用python的多处理模块，让每个进程处理一大片大图像并在处理完成后加入结果（这可能应该在POSIX系统上执行？）2）由于NumPy支持OpenMP而OpenCV使用NumPy，我可以将多处理留给NumPy吗？

I'm using Python 3.4.3 and OpenCV 3.0.0 to process (applying various filters to) a very large image (80,000 x 60,000) in memory and I'd like to use multiple CPU cores to improve performance. After some reading, I arrived at two possible method : 1) Use python's multiprocessing module, let each process deal with a slice of the large image and join the results after processing is done (And this probably should be performed on POSIX system?) 2) Since NumPy supports OpenMP and OpenCV uses NumPy, I can just leave the multiprocessing to NumPy?

所以我的问题是：

哪一个会更好？ （如果它们看起来不合理，那么可能的方法是什么？）

Which one will be a better solution? (If they don't seem reasonable, what would be a possible approach? )

如果选项2是好的，我应该用OpenMP构建NumPy和OpenCV吗？我如何实际进行多处理？ （我真的找不到有用的指示..）

If Option 2 is good, should I build both NumPy and OpenCV with OpenMP ? How would I actually make multi-processing happen? ( I couldn't really find useful instruction..)

推荐答案

在阅读了一些SO帖子之后，我想出了一个在Python3中使用 OpenCV 的方法，使用多处理。我建议在linux上这样做，因为根据这篇文章，只要内容未更改，生成的进程就与父进程共享内存。这是一个最小的例子：

After reading some SO posts, I've come up with a way to use OpenCV in Python3 with multiprocessing. I recommend doing this on linux, because according to this post, spawned processes share memory with their parent as long as the content is not changed. Here's a minimal example:

import cv2
import multiprocessing as mp
import numpy as np
import psutil

img = cv2.imread('test.tiff', cv2.IMREAD_ANYDEPTH) # here I'm using a indexed 16-bit tiff as an example.
num_processes = 4
kernel_size = 11
tile_size = img.shape[0]/num_processes  # Assuming img.shape[0] is divisible by 4 in this case

output = mp.Queue()

def mp_filter(x, output):
    print(psutil.virtual_memory())  # monitor memory usage
    output.put(x, cv2.GaussianBlur(img[img.shape[0]/num_processes*x:img.shape[0]/num_processes*(x+1), :], 
               (kernel_size, kernel_size), kernel_size/5))
    # note that you actually have to process a slightly larger block and leave out the border.

if __name__ == 'main':
    processes = [mp.Processes(target=mp_slice, args=(x, output)) for x in range(num_processes)]

    for p in processes:
        p.start()

    result = []
    for ii in range(num_processes):
        result.append(output.get(True))

    for p in processes:
        p.join()

而不是使用 Queue ，另一种从流程中收集结果的方法是通过多处理创建共享数组模块。 （必须导入 ctypes ）

Instead of using Queue, another way to collect the result from the processes is to create a shared array through multiprocessing module. (Has to import ctypes)

result = mp.Array(ctypes.c_uint16, img.shape[0]*img.shape[1], lock = False)

然后，假设没有重叠，每个进程可以写入数组的不同部分。但是，创建一个大的 \\ tmp.Array 的速度非常慢。这实际上违背了加速操作的目的。因此，只有当与总计算时间相比，增加的时间不多时才使用它。这个数组可以通过以下方式变成一个numpy数组：

Then each process can write to different portions of the array assuming there is no overlap. Creating a large mp.Array is surprisingly slow, however. This actually defies the purpose of speeding up the operation. So use it only when the added time is not much when compared with total computation time. This array can be turned into a numpy array by :

result_np = np.frombuffer(result, dtypye=ctypes.c_uint16)

这篇关于如何使用Python和OpenCV进行多处理？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！