与DASK阵列内存错误 [英] Memory error with dask array

问题描述

我实现神经网络的输入和输出矩阵是非常大的，所以我使用DASK阵列来存储他们。

I am implementing Neural Network whose input and output matrices are very large, so I am using dask arrays for storing them.

X 是输入矩阵32000点¯x7500和是是相同尺寸的输出矩阵。

X is input matrix of 32000 x 7500 and y is output matrix of same dimension.

下面是具有神经网络code 1隐层：

Below is neural network code having 1 hidden layer:

class Neural_Network(object):
    def __init__(self,i,j,k):
        #define hyperparameters
        self.inputLayerSize = i
        self.outputLayerSize = j
        self.hiddenLayerSize = k
        #weights
        self.W1 = da.random.normal(0.5,0.5,size =(self.inputLayerSize,self.hiddenLayerSize),chunks=(1000,1000))
        self.W2 = da.random.normal(0.5,0.5,size =(self.hiddenLayerSize,self.outputLayerSize),chunks=(1000,1000))
        self.W1 = self.W1.astype('float96')
        self.W2 = self.W2.astype('float96')

    def forward(self,X):
        self.z2 = X.dot(self.W1)
        self.a2 = self.z2.map_blocks(self.sigmoid)
        self.z3 = self.a2.dot(self.W2)
        yhat = self.z3.map_blocks(self.sigmoid)
        return yhat

    def exp(z):
        return np.exp(z)

    def sigmoid(self,z):
        #sigmoid function
##        return 1/(1+np.exp(-z))
        return 1/(1+(-z).map_blocks(self.exp))

    def sigmoidprime(self,z):
        ez = (-z).map_blocks(self.exp)
        return ez/(1+ez**2)

    def costFunction (self,X,y):
        self.yHat = self.forward(X)
        return 1/2*sum((y-self.yHat)**2)

    def costFunctionPrime (self,X,y):
        self.yHat = self.forward(X)
        self.error = -(y - self.yHat)
        self.delta3 = self.error*self.z3.map_blocks(self.sigmoidprime)
        dJdW2 = self.a2.transpose().dot(self.delta3)
        self.delta2 = self.delta3.dot(self.W2.transpose())*self.z2.map_blocks(self.sigmoidprime)
        dJdW1 = X.transpose().dot(self.delta2)
        return dJdW1 , dJdW2

现在我尽量减少，如下运行成本：

Now I try to reduce cost of function as below:

>>> n = Neural_Network(7420,7420,5000)
>>> for i in range(0,500):
    cost1,cost2 = n.costFunctionPrime(X,y)
    n.W1 = n.W1 -3*cost1
    n.W2 = n.W2 -3*cost2
    if i%5==0:
        print (i*100/500,'%')

但是，当 I 达到约120它给我的错误：

But when i reaches around 120 it gives me error:

    File "<pyshell#127>", line 3, in <module>
    n.W1 = n.W1 -3*cost1
  File "c:\python34\lib\site-packages\dask\array\core.py", line 1109, in __sub__
    return elemwise(operator.sub, self, other)
  File "c:\python34\lib\site-packages\dask\array\core.py", line 2132, in elemwise
    dtype=dt, name=name)
  File "c:\python34\lib\site-packages\dask\array\core.py", line 1659, in atop
    return Array(merge(dsk, *dsks), out, chunks, dtype=dtype)
  File "c:\python34\lib\site-packages\toolz\functoolz.py", line 219, in __call__
    return self._partial(*args, **kwargs)
  File "c:\python34\lib\site-packages\toolz\curried\exceptions.py", line 20, in merge
    return toolz.merge(*dicts, **kwargs)
  File "c:\python34\lib\site-packages\toolz\dicttoolz.py", line 39, in merge
    rv.update(d)
MemoryError

这也给了的MemoryError 当我这样做 nn.W1.compute（）

推荐答案

这看起来像它的失败，同时建立图表，而不是计算中。两件事情浮现在脑海中：

This looks like its failing while building the graph, not during computation. Two things come to mind:

你的循环可能会倾倒数以百万计的任务到任务图中的每个迭代。每个任务可能占用类似100B为1kB的。当这些加起来，他们可以很容易地使您的机器。

Each iteration of your for loop may be dumping millions of tasks into the task graph. Each task probably takes up something like 100B to 1kB. When these add up they can easily overwhelm your machine.

在一个典型的深度学习库，像Theano，您将使用扫描操作这样的事情。 Dask.array没有这样的操作。

In a typical deep learning library, like Theano, you would use a scan operation for something like this. Dask.array has no such operation.

您拨打map_blocks上本身调用map_blocks的功能。

You call map_blocks on a function that itself calls map_blocks.

self.delta2 = self.delta3.dot(self.W2.transpose())*self.z2.map_blocks(self.sigmoidprime)

def sigmoidprime(self,z):
    ez = (-z).map_blocks(self.exp)
    return ez/(1+ez**2)

相反，你可能只是做一个乙状结肠主要功能

Instead you might just make a sigmoid prime function

def sigmoidprime(z):
    ez = np.exp(-z)
    return ez / (1 + ez ** 2)

然后映射功能

self.z2.map_blocks(sigmoidprime)

深学习是棘手

一般来说，做深学以及经常需要专业化。设计一般做这口井的库不是有原因的一般用途。通用库，如dask.array的可能的是有用的，但可能永远达不到一个图书馆的流畅运行像Theano。

Deep learning is tricky

Generally speaking, doing deep learning well often requires specialization. The libraries designed to do this well generally aren't general purpose for a reason. A general purpose library, like dask.array might be useful but will probably never reach the smooth operation of a library like Theano.

您可以尝试建立一个函数，只需一步。它将从磁盘读取，做所有的点产品，调换，和正常的计算，然后将明确存储到磁盘上的数据集。然后，您可以调用这个函数多次。即使是这样，我不相信背后dask.array的调度策略可以做到这一点。

You might try building a function that takes just one step. It would read from disk, do all of your dot products, transposes, and normal computations, and would then store explicitly into an on-disk dataset. You would then call this function many times. Even then I'm not convinced that the scheduling policies behind dask.array could do this well.

这篇关于与DASK阵列内存错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！