如何使压差取反以补偿压差的影响并保持期望值不变? [英] How inverting the dropout compensates the effect of dropout and keeps expected values unchanged?

问题描述

我正在从deeplearning.ai课程中学习神经网络中的正则化.教授说，在辍学正则化中，如果应用了辍学，则计算出的激活值将小于未应用辍学(测试时)的激活值.因此，我们需要调整激活次数，以使测试阶段更简单.

I'm learning regularization in Neural networks from deeplearning.ai course. Here in dropout regularization, the professor says that if dropout is applied, the calculated activation values will be smaller then when the dropout is not applied (while testing). So we need to scale the activations in order to keep the testing phase simpler.

我了解这个事实，但是我不知道如何进行缩放.这是一个用于实现反向辍学的代码示例.

I understood this fact, but I don't understand how scaling is done. Here is a code sample which is used to implement inverted dropout.

keep_prob = 0.8   # 0 <= keep_prob <= 1
l = 3  # this code is only for layer 3
# the generated number that are less than 0.8 will be dropped. 80% stay, 20% dropped
d3 = np.random.rand(a[l].shape[0], a[l].shape[1]) < keep_prob

a3 = np.multiply(a3,d3)   # keep only the values in d3

# increase a3 to not reduce the expected value of output
# (ensures that the expected value of a3 remains the same) - to solve the scaling problem
a3 = a3 / keep_prob  

在上面的代码中，为什么将激活除以0.8或将节点保留在层中的概率(keep_prob)?任何数值示例都会有所帮助.

In the above code, why the activations are divided by 0.8 or the probability of keeping a node in a layer (keep_prob)? Any numerical example will help.

推荐答案

我花了一些时间了解倒置的辍学现象，才得到了答案.这是直觉:

I got the answer by myself after spending some time understanding the inverted dropout. Here is the intuition:

我们以keep_prob的概率将神经元保存在任何层中.假设kepp_prob = 0.6.这意味着要关闭任何一层中40％的神经元.如果在关闭40％的神经元之前该层的原始输出为x，则在应用40％的退出后，该输出将减少 0.4 * x.所以现在是x - 0.4x = 0.6x.

We are preserving the neurons in any layer with the probability keep_prob. Let's say kepp_prob = 0.6. This means to shut down 40% of the neurons in any layer. If the original output of the layer before shutting down 40% of neurons was x, then after applying 40% dropout, it'll be reduced by 0.4 * x. So now it will be x - 0.4x = 0.6x.

要保持原始输出(期望值)，我们需要将输出除以keep_prob(或此处的0.6).

To maintain the original output (expected value), we need to divide the output by keep_prob (or 0.6 here).

这篇关于如何使压差取反以补偿压差的影响并保持期望值不变?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！