XOR 神经网络误差在训练期间停止减少 [英] XOR neural network error stops decreasing during training

问题描述

我正在使用随机梯度下降通过反向传播训练 XOR 神经网络.神经网络的权重被初始化为 -0.5 到 0.5 之间的随机值.神经网络在大约 80% 的时间内成功地自我训练.然而，有时它在反向传播时会卡住".通过卡住"，我的意思是我开始看到错误纠正率下降.例如，在成功的训练过程中，总误差随着网络学习而迅速减少，如下所示:

I'm training a XOR neural network via back-propagation using stochastic gradient descent. The weights of the neural network are initialized to random values between -0.5 and 0.5. The neural network successfully trains itself around 80% of the time. However sometimes it gets "stuck" while backpropagating. By "stuck", I mean that I start seeing a decreasing rate of error correction. For example, during a successful training, the total error decreases rather quickly as the network learns, like so:

...
...
Total error for this training set: 0.0010008071327708653
Total error for this training set: 0.001000750550254843
Total error for this training set: 0.001000693973929822
Total error for this training set: 0.0010006374037948094
Total error for this training set: 0.0010005808398488103
Total error for this training set: 0.0010005242820908169
Total error for this training set: 0.0010004677305198344
Total error for this training set: 0.0010004111851348654
Total error for this training set: 0.0010003546459349181
Total error for this training set: 0.0010002981129189812
Total error for this training set: 0.0010002415860860656
Total error for this training set: 0.0010001850654351723
Total error for this training set: 0.001000128550965301
Total error for this training set: 0.0010000720426754587
Total error for this training set: 0.0010000155405646494
Total error for this training set: 9.99959044631871E-4

Testing trained XOR neural network
0 XOR 0: 0.023956746649767453
0 XOR 1: 0.9736079194769579
1 XOR 0: 0.9735670067093437
1 XOR 1: 0.045068688874314006

然而，当它卡住时，总的错误正在减少，但似乎是在下降:

However when it gets stuck, the total errors are decreasing, but it seems to be at a decreasing rate:

...
...
Total error for this training set: 0.12325486644721295
Total error for this training set: 0.12325486642503929
Total error for this training set: 0.12325486640286581
Total error for this training set: 0.12325486638069229
Total error for this training set: 0.12325486635851894
Total error for this training set: 0.12325486633634561
Total error for this training set: 0.1232548663141723
Total error for this training set: 0.12325486629199914
Total error for this training set: 0.12325486626982587
Total error for this training set: 0.1232548662476525
Total error for this training set: 0.12325486622547954
Total error for this training set: 0.12325486620330656
Total error for this training set: 0.12325486618113349
Total error for this training set: 0.12325486615896045
Total error for this training set: 0.12325486613678775
Total error for this training set: 0.12325486611461482
Total error for this training set: 0.1232548660924418
Total error for this training set: 0.12325486607026936
Total error for this training set: 0.12325486604809655
Total error for this training set: 0.12325486602592373
Total error for this training set: 0.12325486600375107
Total error for this training set: 0.12325486598157878
Total error for this training set: 0.12325486595940628
Total error for this training set: 0.1232548659372337
Total error for this training set: 0.12325486591506139
Total error for this training set: 0.12325486589288918
Total error for this training set: 0.12325486587071677
Total error for this training set: 0.12325486584854453

在我阅读神经网络时，我遇到了关于局部最小值和全局最小值的讨论，以及神经网络如何不真正知道"它应该走向哪个最小值.

While I was reading up on neural networks I came across a discussion on local minimas and global minimas and how neural networks don't really "know" which minima its supposed to be going towards.

我的网络是否陷入局部最小值而不是全局最小值?

Is my network getting stuck in a local minima instead of a global minima?

推荐答案

是的，神经网络可能会陷入局部最小值，这取决于错误面.然而这个摘要表明异或问题的误差面中没有局部最小值.但是，我无法获得全文，因此我无法验证作者为证明这一点所做的工作以及它如何适用于您的问题.

Yes, neural networks can get stuck in local minima, depending on the error surface. However this abstract suggests that there are no local minima in the error surface of the XOR problem. However I cannot get to the full text, so I cannot verify what the authors did to proove this and how it applies to your problem.

可能还有其他因素导致此问题.例如，如果您在某个陡峭的山谷下非常快地下降，如果您只使用一阶梯度下降，您可能会到达相反的斜坡并一直来回反弹.您还可以尝试在每次迭代时给出所有权重的平均变化，以测试您是否真的有一个卡住"的网络，或者更确切地说是一个刚刚进入极限循环的网络.

There also might be other factors leading to this problem. For example if you descend very fast at some steep valley, if you just use a first order gradient descent, you might get to the opposite slope and bounce back and forth all the time. You could try also giving the average change over all weights at each iteration, to test if you realy have a "stuck" network, or rather one, which just has run into a limit cycle.

你应该首先尝试摆弄你的参数(学习率、动量，如果你实现了它等).如果您可以通过更改参数使问题消失，那么您的算法可能没问题.

You should first try fiddling with your parameters (learning rate, momentum if you implemented it etc). If you can make the problem go away, by changing parameters, your algorithm is probably ok.

这篇关于XOR 神经网络误差在训练期间停止减少的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！