softmax函数说明的导数 [英] Derivative of a softmax function explanation

问题描述

我正在尝试为softmax计算激活函数的导数.我发现了这一点: https://math.stackexchange.com/questions/945871/derivative- of-softmax-loss-function 似乎没有人对我们如何获得i = j和i！= j的答案给出正确的推导.有人可以解释一下！当涉及求和时，如softmax激活函数的分母中，我对导数感到困惑.

I am trying to compute the derivative of the activation function for softmax. I found this : https://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function nobody seems to give the proper derivation for how we would get the answers for i=j and i!= j. Could someone please explain this! I am confused with the derivatives when a summation is involved as in the denominator for the softmax activation function.

推荐答案

和的导数是导数的和，即:

The derivative of a sum is the sum of the derivatives, ie:

    d(f1 + f2 + f3 + f4)/dx = df1/dx + df2/dx + df3/dx + df4/dx

要得出相对于o_i的p_j的导数，我们开始于:

To derive the derivatives of p_j with respect to o_i we start with:

    d_i(p_j) = d_i(exp(o_j) / Sum_k(exp(o_k)))

我决定使用d_i作为与o_i有关的派生词，以使其更易于阅读. 使用产品规则，我们得到:

I decided to use d_i for the derivative with respect to o_i to make this easier to read. Using the product rule we get:

     d_i(exp(o_j)) / Sum_k(exp(o_k)) + exp(o_j) * d_i(1/Sum_k(exp(o_k)))

看第一项，如果i != j，则导数将为0，可以用

Looking at the first term, the derivative will be 0 if i != j, this can be represented with a delta function which I will call D_ij. This gives (for the first term):

    = D_ij * exp(o_j) / Sum_k(exp(o_k))

这只是我们原来的函数乘以D_ij

Which is just our original function multiplied by D_ij

    = D_ij * p_j

对于第二项，当我们分别导出和的每个元素时，唯一的非零项将是i = k时，这给了我们(不要忘记幂定律，因为和在分母中)

For the second term, when we derive each element of the sum individually, the only non-zero term will be when i = k, this gives us (not forgetting the power rule because the sum is in the denominator)

    = -exp(o_j) * Sum_k(d_i(exp(o_k)) / Sum_k(exp(o_k))^2
    = -exp(o_j) * exp(o_i) / Sum_k(exp(o_k))^2
    = -(exp(o_j) / Sum_k(exp(o_k))) * (exp(o_j) / Sum_k(exp(o_k)))
    = -p_j * p_i

将两者放在一起，我们得到了令人惊讶的简单公式:

Putting the two together we get the surprisingly simple formula:

    D_ij * p_j - p_j * p_i

如果您确实需要，我们可以将其分为i = j和i != j两种情况:

If you really want we can split it into i = j and i != j cases:

    i = j: D_ii * p_i - p_i * p_i = p_i - p_i * p_i = p_i * (1 - p_i)

    i != j: D_ij * p_i - p_i * p_j = -p_i * p_j

这是我们的答案.

这篇关于softmax函数说明的导数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！