numpy矩阵的并行循环 [英] Parallel for loop over numpy matrix

问题描述

我正在查看 joblib 示例，但我不知道该怎么做在矩阵上进行并行for循环.我正在计算矩阵行之间的成对距离度量.所以我在做:

I am looking at the joblib examples but I can't figure out how to do a parallel for loop over a matrix. I am computing a pairwise distance metric between the rows of a matrix. So I was doing:

N, _ = data.shape
upper_triangle = [(i, j) for i in range(N) for j in range(i + 1, N)]
dist_mat = np.zeros((N,N))  

for (i, j) in upper_triangle:
    dist_mat[i,j] = dist_fun(data[i], data[j])
    dist_mat[j,i] = dist_mat[i,j]

其中，dist_fun采用两个向量并计算距离.由于对dist_fun的调用可以相互独立，因此如何使此循环并行进行.

where dist_fun takes two vectors and computes a distance. How can I make this loop parallel since calls to dist_fun can be made independent of each other.

我正在使用的距离函数是 fastdtw ，它不是太快了.因此，我认为 确实如此 希望对此进行并行化.使用:

The distance function I am using is fastdtw which is not so fast. So I think really do want to parallelize this. Using:

dist_mat = pdist(data, lambda x,y : fastdtw(x,y, dist=euclidean)[0])

我得到58.1084秒的执行时间，并使用:

I get an execution time of 58.1084 secs, and using:

dist_mat = np.zeros((N,N))
for (i,j), _ in np.ndenumerate(dist_mat):
    dist_mat[i,j], _ = fastdtw(data[i,:], timeseries[j,:], dist=euclidean)

我得到116.36秒并使用:

I get 116.36 seconds and using:

upper_triangle = [(i,j) for i in range(N) for j in range(i+1, N)]
dist_mat = np.zeros((N,N))

for (i,j) in upper_triangle:
    dist_mat[i,j], _ = fastdtw(data[i,:], data[j,:], dist=euclidean)
    dist_mat[j,i] = dist_mat[i,j]

我得到55.62秒.在这里N=33. scipy是否自动利用所有可用的内核?

I get 55.62 secs. Here N=33. Does scipy automatically makes use of all available cores?

我想我已经找到了使用 multiprocessing 程序包，但是在我发布我认为可行的内容之前，我将不回答#strong> joblib 的问题.

I think I have found a work around using the multiprocessing package, but I will leave the question un-answered for the joblib folks to respond before I post what I think works.

推荐答案

可以使用 multiprocessing 模块:

This can be done as follows using the multiprocessing module:

import numpy as np
from fastdtw import fastdtw
import multiprocessing as mp
from scipy.spatial.distance import squareform, euclidean
from functools import partial

# Create simulated data matrix
data = np.random.random((33,300))

N, _ = data.shape
upper_triangle = [(i,j) for i in range(N) for j in range(i+1, N)]

with mp.Pool(processes=4) as pool:
    result = pool.starmap(partial(fastdtw, dist=euclidean), [(data[i], data[j]) for (i,j) in upper_triangle])

dist_mat = squareform([item[0] for item in result])

使用 timeit 在这是没有显式并行化的一半时间.

which is half the time without explicit parallelization.

还:

作为使用 fastdtw 软件包的任何人的将来参考.如链接示例中所示，从scipy.spatial.distance导入距离函数并调用fastdtw比使用fastdtw(x,y,dist=2)慢得多.结果似乎与我相似，并且使用pdist的执行时间(不求助于并行化)不到一秒钟.

As a future reference for anyone using the fastdtw package. Importing the distance functions from scipy.spatial.distance and calling fastdtw as shown in the example on the link is much slower than just using: fastdtw(x,y,dist=2). The results seem similar to me and the execution time using pdist (without resorting to parallelization) is under a second.

这篇关于numpy矩阵的并行循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！