使用RELU的神经网络反向传播 [英] Neural network backpropagation with RELU

问题描述

我正在尝试使用RELU实现神经网络.

I am trying to implement neural network with RELU.

输入层-> 1个隐藏层-> relu->输出层-> softmax层

input layer -> 1 hidden layer -> relu -> output layer -> softmax layer

以上是我的神经网络的体系结构. 我对这个relu的反向传播感到困惑. 对于RELU的导数，如果x <= 0，则输出为0. 如果x> 0，则输出为1. 因此，当您计算梯度时，是否意味着如果x< = 0，我就杀​​死了梯度体面的人?

Above is the architecture of my neural network. I am confused about backpropagation of this relu. For derivative of RELU, if x <= 0, output is 0. if x > 0, output is 1. So when you calculate the gradient, does that mean I kill gradient decent if x<=0?

有人可以逐步解释我的神经网络架构的反向传播吗?

Can someone explain the backpropagation of my neural network architecture 'step by step'?

推荐答案

如果x< == 0，则输出为0.如果x> 0，则输出为1

if x <= 0, output is 0. if x > 0, output is 1

ReLU函数定义为:对于x> 0，输出为x，即 f(x)= max(0，x)

The ReLU function is defined as: For x > 0 the output is x, i.e. f(x) = max(0,x)

所以对于导数f'(x)实际上是:

So for the derivative f '(x) it's actually:

如果x< 0，输出为0.如果x> 0，则输出为1.

if x < 0, output is 0. if x > 0, output is 1.

未定义导数f'(0).因此通常将其设置为0，或者将激活函数修改为f(x)= max(e，x)(对于较小的e).

The derivative f '(0) is not defined. So it's usually set to 0 or you modify the activation function to be f(x) = max(e,x) for a small e.

通常:ReLU是使用整流器激活功能的单元.这意味着它的工作原理与其他任何隐藏层完全相同，但除了tanh(x)，sigmoid(x)或您使用的任何激活方法外，您将改为使用f(x)= max(0，x).

Generally: A ReLU is a unit that uses the rectifier activation function. That means it works exactly like any other hidden layer but except tanh(x), sigmoid(x) or whatever activation you use, you'll instead use f(x) = max(0,x).

如果您已经为具有S形激活功能的多层网络编写了代码，则字面上的变化是1行.关于正向传播或反向传播，在算法上没有任何改变.如果还没有运行更简单的模型，请先回过头来开始.否则，您的问题不是真正关于ReLU，而是关于整体实现NN.

If you have written code for a working multilayer network with sigmoid activation it's literally 1 line of change. Nothing about forward- or back-propagation changes algorithmically. If you haven't got the simpler model working yet, go back and start with that first. Otherwise your question isn't really about ReLUs but about implementing a NN as a whole.

这篇关于使用RELU的神经网络反向传播的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！