numpy:有效执行数组的复杂整形 [英] numpy: efficient execution of a complex reshape of an array

问题描述

我正在将供应商提供的大型二进制数组读入2D numpy数组tempfid(M，N)

I am reading a vendor-provided large binary array into a 2D numpy array tempfid(M, N)

# load data
data=numpy.fromfile(file=dirname+'/fid', dtype=numpy.dtype('i4'))

# convert to complex data
fid=data[::2]+1j*data[1::2]

tempfid=fid.reshape(I*J*K, N)

，然后我需要使用非平凡的索引映射将其整形为4D(N，I，J，K)数组.我通过以下行的for循环来做到这一点:

and then I need to reshape it into a 4D array useful4d(N,I,J,K) using non-trivial mappings for the indices. I do this with a for loop along the following lines:

for idx in range(M):
    i=f1(idx) # f1, f2, and f3 are functions involving / and % as well as some lookups
    j=f2(idx)
    k=f3(idx)
    newfid[:,i,j,k] = tempfid[idx,:] #SLOW! CAN WE IMPROVE THIS?

转换为复数需要33％的时间，而复制这些M片将花费其余66％的时间.无论我是在所示的循环中一个接一个地执行此操作，还是通过对操作进行numpy.vectorize并将其应用于arange(M)来进行计算，索引的计算都是快速的.

Converting to complex takes 33% of the time while the copying of these slices M slices takes the remaining 66%. Calculating the indices is fast irrespective of whether I do this one by one in a loop as shown or by numpy.vectorizing the operation and applying it to an arange(M).

有没有办法加快速度?感谢您提供有关更高效切片，复制(或不复制)等方面的帮助.

Is there a way to speed this up? Any help on more efficient slicing, copying (or not) etc appreciated.

正如对问题的回答所了解的，将交错的NumPy整数数组转换为complex64的最快方法是什么?" 转换为如果改用视图，则复杂度可以提高6倍:

As learned in the answer to question "What's the fastest way to convert an interleaved NumPy integer array to complex64?" the conversion to complex can be sped up by a factor of 6 if a view is used instead:

 fid = data.astype(numpy.float32).view(numpy.complex64)

推荐答案

idx = numpy.arange(M)
i = numpy.vectorize(f1)(idx)
j = numpy.vectorize(f2)(idx)
k = numpy.vectorize(f3)(idx)

# you can index arrays with other arrays
# that lets you specify this operation in one line.    
newfid[:, i,j,k] = tempfid.T

我从没用过numpy的vectorize.向量化只是意味着numpy将多次调用您的python函数.为了提高速度，您需要使用数组运算，如我在此处显示的那样，并且您过去曾获得过复数.

I've never used numpy's vectorize. Vectorize just means that numpy will call your python function multiple times. In order to get speed, you need use array operations like the one I showed here and you used to get complex numbers.

编辑

问题是大小128的尺寸首先在newfid中，但最后在tempfid中.这很容易通过使用.T进行转置来实现.

The problem is that the dimension of size 128 was first in newfid, but last in tempfid. This is easily by using .T which takes the transpose.

这篇关于numpy:有效执行数组的复杂整形的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！