归一化数据饱和度错误的神经网络 [英] Neural Network with tanh wrong saturation with normalized data

问题描述

我正在使用由4个输入神经元组成的神经网络，1个由20个神经元组成的隐藏层和7个神经元输出层.

I'm using a neural network made of 4 input neurons, 1 hidden layer made of 20 neurons and a 7 neuron output layer.

我正在尝试针对bcd到7段算法进行训练.我的数据归一化为0是-1和1是1.

I'm trying to train it for a bcd to 7 segment algorithm. My data is normalized 0 is -1 and 1 is 1.

发生输出错误评估时，神经元会饱和.如果所需的输出为1，而实际输出为-1，则错误为1-(-1)= 2.

When the output error evaluation happens, the neuron saturates wrong. If the desired output is 1 and the real output is -1, the error is 1-(-1)= 2.

当我将其与激活函数error*(1-output)*(1+output)的导数相乘时，由于2*(1-(-1)*(1-1)，错误几乎变为0.

When I multiply it by the derivative of the activation function error*(1-output)*(1+output), the error becomes almost 0 Because of 2*(1-(-1)*(1-1).

如何避免出现饱和错误?

How can I avoid this saturation error?

推荐答案

激活函数渐近点的饱和度是神经网络的常见问题.如果您看一下函数的图形，就不会感到惊讶:它们几乎是平坦的，这意味着一阶导数是(几乎)0.网络无法再学习了.

Saturation at the asymptotes of of the activation function is a common problem with neural networks. If you look at a graph of the function, it doesn't surprise: They are almost flat, meaning that the first derivative is (almost) 0. The network cannot learn any more.

一个简单的解决方案是扩展激活功能，以避免出现此问题.例如，对于tanh()激活函数(我最喜欢的)，建议在所需输出位于{-1，1}中时使用以下激活函数:

A simple solution is to scale the activation function to avoid this problem. For example, with tanh() activation function (my favorite), it is recommended to use the following activation function when the desired output is in {-1, 1}:

f(x) = 1.7159 * tanh( 2/3 * x)  

因此，导数为

f'(x) = 1.14393 * (1- tanh( 2/3 * x))  

这将迫使梯度进入最非线性的值范围，并加快学习速度.对于所有详细信息，我建议阅读Yann LeCun的出色论文有效的反向支持. 对于tanh()激活函数，该错误将计算为

This will force the gradients into the most non-linear value range and speed up the learning. For all the details I recommend reading Yann LeCun's great paper Efficient Back-Prop. In the case of tanh() activation function, the error would be calculated as

error = 2/3 * (1.7159 - output^2) * (teacher - output)

这篇关于归一化数据饱和度错误的神经网络的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！