Python多处理全局numpy数组 [英] Python multiprocessing global numpy arrays

问题描述

我有以下脚本:

max_number = 100000
minimums = np.full((max_number), np.inf, dtype=np.float32)
data = np.zeros((max_number, 128, 128, 128), dtype=np.uint8)

if __name__ == '__main__':
    main()

def worker(array, start, end):

    for in_idx in range(start, end):
        value = data[start:end][in_idx] # compute something using this array
        minimums[in_idx] = value

def main():

    jobs = []
    num_jobs = 5
    for i in range(num_jobs):
        start = int(i * (1000 / num_jobs))
        end = int(start + (1000 / num_jobs))

        p = multiprocessing.Process(name=('worker_' + str(i)), target=worker, args=(start, end))
        jobs.append(p)
        p.start()

    for proc in jobs:
        proc.join()
    print(jobs)

如何确保numpy数组是全局的，并且每个工作人员都可以访问?每个工作人员使用numpy数组的不同部分

How can I ensure that the numpy array is global and can be accessed by each worker? Each worker uses a different part of the numpy array

推荐答案

import numpy as np
import multiprocessing as mp

ar = np.zeros((5,5))

def callback_function(result):
    x,y,data = result
    ar[x,y] = data

def worker(num):
    data = ar[num,num]+3
    return num, num, data

def apply_async_with_callback():
    pool = mp.Pool(processes=5)
    for i in range(5):
        pool.apply_async(worker, args = (i, ), callback = callback_function)
    pool.close()
    pool.join()
    print "Multiprocessing done!"

if __name__ == '__main__':
    ar = np.ones((5,5)) #This will be used, as local scope comes before global scope
    apply_async_with_callback()

说明:您可以设置数据数组以及工作程序和回调函数.池中的多个进程设置了许多独立的工作程序，其中每个工作程序可以执行多个任务.回调将结果写回到数组.

Explanation: You set up your data array and your workers and callback functions. The number of processes in the pool set up a number of independent workers, where each worker can do more than one task. The callback writes the result back to the array.

__name__=='__main__'防止在每次导入时运行以下行.

The __name__=='__main__' protects the following line from being run at each import.

这篇关于Python多处理全局numpy数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！