Cython memoryview转置:TypeError [英] Cython memoryview transpose: Typeerror
问题描述
我正在尝试使用python开发一个小型的卷积神经网络框架. 卷积节点的代码已经(缓慢地)工作了，我想加快速度. 热点是卷积滤波器在图像上移动的循环.我选择使用cython来加速这些循环.
I'm trying to develop a small Convolutional Neural Network framework with python. The code for the convolutional node already works (slowly) and I would like to speed it up. The hotspots are the loops where the convolutional filter is moved across the image. I chose to use cython to speed up those loops.
明显的小注释，所有局部变量的cdef并删除了boundscheck，几乎减少了我运行时的10％.根据我在网上阅读的内容，cython应该已经能够做到这一点，这对我来说似乎很奇怪.
The obvious small annotations, cdef for all local variables and removing boundscheck, shaved hardly 10% off of my runtime. That seemed strange to me, based on what I read online, cython should already be able to do its magic.
不幸的是，代码位于类中，并且严重依赖于该类的属性.我决定将其转换为cdef类.这意味着必须使用cdef声明所有类属性.显然，cython不支持numpy数组，因此我将所有numpy数组声明为double[:,:,...]
Unfortunately the code is inside a class and relies heavily on the properties of that class. I decided to convert it into a cdef class. This means that all class attributes have to be declared with cdef. Apparently cython doesn't support numpy arrays, so I declared all numpy arrays as double[:,:,...]
到目前为止，所有单元测试都通过了，代码可以正常工作.现在，对.pyd的编译(我在Windows下工作)仍然有效.但是运行代码会产生Typeerror:
So far the code worked fine, all unittests passing. Now the compilation to .pyd (I'm working under windows) still works. But running the code creates a Typeerror:
TypeError:只有length-1个数组可以转换为Python标量
TypeError: only length-1 arrays can be converted to Python scalars
这是一些代码.这是我的卷积节点的整个前向方法，可能太多而且不易阅读.您可能只需要最后一行.那是发生错误的地方:
Here is some code. This is the entire forward method of my convolutional node, which might be too much and not easily readable. You probably only need the very last line. That's were the error happens:
@cython.boundscheck(False)
@cython.nonecheck(False)
def forward(self):
# im2col: x -> in_cols
# padding
cdef np.ndarray[DTYPE_t, ndim=4] x_padded = np.zeros((self.batch_size, self.in_colors, self.in_width + self.padding*2, self.in_height + self.padding*2))
if self.padding>0:
x_padded[:, :, self.padding:self.in_width+self.padding, self.padding:self.in_height+self.padding] = self.x
else:
x_padded[:]=self.x
# allocating new field
cdef np.ndarray[DTYPE_t, ndim=4] rec_fields = np.empty((self.filter_size**2* self.in_colors, self.batch_size, self.out_width, self.out_height))
# copying receptive fields
cdef int w,h
for w, h in np.ndindex((self.out_width, self.out_height)):
rec_fields[:, :, w, h] = x_padded[:, :, w*self.stride:w*self.stride + self.filter_size, h*self.stride:h*self.stride + self.filter_size] \
.reshape((self.batch_size, self.filter_size**2* self.in_colors)) \
.T
self.in_cols = rec_fields.reshape((self.filter_size**2 * self.in_colors, self.batch_size * self.out_width * self.out_height))
# linear node: in_cols -> out_cols
cdef np.ndarray[DTYPE_t, ndim=2] out_cols=np.dot(self.W,self.in_cols)+self.b
# col2im: out_cols -> out_image -> y
cdef np.ndarray[DTYPE_t, ndim=4] out_image = out_cols.reshape((self.out_colors, self.batch_size, self.out_width, self.out_height))
self.y[:] = out_image.transpose(1, 0, 2, 3)
该移调的最后一次调用在异常中标记.我无法解释这一点.转置后，内存视图的行为是否有所不同?
This last call to transpose is marked in the exception. I can't explain this. Do memoryviews behave differently when transposed ?
更新:
我确定尺寸定义正确.如果尺寸不匹配，则会产生另一个运行时错误.现在无法检查，但是有点像得到4角，预期2角". 我必须说，cython的类型系统给我留下了深刻的印象. python异常中的此类运行时类型信息非常有用. 可悲的是，它并不能解释为什么上述转置会失败.
I'm sure that the dimensions are defined correctly. If there is a dimension mismatch, it produces a different runtime error. Can't check right now, but it was something like "got 4-dim, expected 2-dim". I've got to say that I'm extremely impressed by the type system of cython. This kind of runtime type information in a python exception is rather useful. Sadly it doesn't explain why the transpose above fails.
更新:
数组有一些复杂之处:不能覆盖它们，只能用作引用.
There's some complication with the arrays: They must not be overwritten, only be used as references.
有点难以解释: 神经网络的核心是一个循环，该循环在网络中的所有节点上连续调用方法forward().
A little difficult to explain: At the core of the neural network is a loop which calls the method forward() on all nodes in the network consecutively.
for node in self.nodes:
node.forward()
在这种方法中，节点查看其输入数据，进行一些计算并写入其输出.它依赖于输入已经包含正确数据的事实.
In this method the node looks at its input data, makes some computations and writes to its output. It relies on the fact that the input already contains the correct data.
对于网络的设置，我以正确的顺序存储节点.我手动连接它们.
For the setup of my network I store the nodes in the right order. And I connect them manually.
node2.x=node1.y
现在如果我写
self.y[:]= data
在node1的正向方法中，node2自动具有正确的输入. 这需要仔细的编程:必须以正确的顺序调用forward方法，并且绝不能覆盖输出，而只能对其进行写入.
in the forward method of node1, node2 automatically has the correct input. This requires careful programming: the forward methods must be called in the right order and the output must never be overwritten, only written to.
替代方案将是一个巨大的结构，其中我存储了每个节点的输出并将这些数据传递给周围.这样会创建大量样板代码，并弄乱了向前和向后的传递.
The alternative would be a huge structure where I store the output of each node and pass this data around. That would create lots of boilerplate code and mess up the forward and backward pass.
更新:
向前的最后几行现在看起来像这样:
the last few lines in forward now look like this:
cdef np.ndarray[DTYPE_t, ndim=4] out_image = out_cols.reshape((self.out_colors, self.batch_size, self.out_width, self.out_height))
cdef double[:,:,:,:] temp
temp=out_image.transpose(1,0,2,3)
self.y[...] = temp
分配给temp失败，并带有相同的TypeError消息.
The assignment to temp fails with the same TypeError message.
推荐答案
self.y[...] = some_array
# or equivalently self.y[:,:,:,:] = some_array
将some_array的副本复制到self.y中，该副本必须已经初始化为正确的大小.如果some_array已经是一个内存视图，它似乎也只能工作(对我来说意义不大，但这似乎是事实).
does a copy of some_array into self.y, which must already be initialised to the right size. It also only seems to work if some_array is already a memoryview (which doesn't hugely make sense to me, but this seems to be the case).
(self.y[:] = some_array仅适用于一维数组)
如果您只想让self.y查看"您想要做的一个numpy数组
If you just want make self.y "look at" a numpy array you just want to do
self.y = some_array
# in your case:
# self.y = out_image.transpose(1, 0, 2, 3)
这很可能适合您的目的！
The chances are that the this is fine for your purposes!
如果您特别热衷于制作副本(可能是因为您已经获取了指向self.y的C指针或类似的东西)，则必须强制将some_array用作内存视图.你会做类似的事情
If you're particularly keen on making a copy (possibly if you've taken a C pointer to self.y or something like that) then you have to force some_array to be a memoryview. You'd do something like
cdef double[:,:,:,:] temporary_view_of_transpose
# temporary_view_of_transpose now "looks at" the memory allocated by transpose
# no square brackets!
temporary_view_of_transpose = out_image.transpose(1, 0, 2, 3)
# data is copied from temporary_view_of_transpose to self.y
self.y[...] = temporary_view_of_transpose # (remembering that self.y must be the correct shape before this assignment).
我同意看到的错误消息没有帮助！
I agree the error message seen is unhelpful!
以下是对我有用的最小完整示例(Cython 0.24，Python 3.5.1，Linux-我无法轻松地在Anaconda上进行测试)在此阶段，我不清楚您的代码有什么不同.
The following is minimum complete example that works for me (Cython 0.24, Python 3.5.1, Linux - I can't easily test on Anaconda) At this stage I'm not clear what's different in your code.
# memview.pyx
cimport numpy as np
import numpy as np
cdef class MemviewClass:
cdef double[:,:,:,:] y
def __init__(self):
self.y = np.zeros((2,3,4,5))
def do_something(self):
cdef np.ndarray[np.float64_t,ndim=4] out_image = np.ones((3,2,4,5))
cdef double[:,:,:,:] temp
temp = out_image.transpose(1,0,2,3)
self.y[...] = temp
def print_y(self):
# just to check it gets changed
print(np.asarray(self.y))
和test_script.py证明它有效:
and test_script.py to show it works:
# use pyximport for ease of testing
import numpy
import pyximport; pyximport.install(setup_args=dict(include_dirs=numpy.get_include()))
import memview
a = memview.MemviewClass()
a.print_y() # prints a big array of 0s
a.do_something()
a.print_y() # prints a big array of 1s
这篇关于Cython memoryview转置:TypeError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
