我的交叉熵函数的实现有什么问题? [英] What is the problem with my implementation of the cross-entropy function?

问题描述

我正在学习神经网络，我想在python中编写函数cross_entropy.定义为

I am learning the neural network and I want to write a function cross_entropy in python. Where it is defined as

其中，N是样本数，k是类数， log 是自然对数，如果样本i在类中，则t_i,j是1 j和0否则，p_i,j是样本i在类j中的预测概率. 为了避免对数出现数值问题，请将预测范围限制在[10^{−12}, 1 − 10^{−12}]范围内.

where N is the number of samples, k is the number of classes, log is the natural logarithm, t_i,j is 1 if sample i is in class j and 0 otherwise, and p_i,j is the predicted probability that sample i is in class j. To avoid numerical issues with logarithm, clip the predictions to [10^{−12}, 1 − 10^{−12}] range.

根据上面的描述，我通过将预测剪裁到[epsilon, 1 − epsilon]范围，然后根据上面的公式计算cross_entropy来写下代码.

According to the above description, I wrote down the codes by clipping the predictions to [epsilon, 1 − epsilon] range, then computing the cross_entropy based on the above formula.

def cross_entropy(predictions, targets, epsilon=1e-12):
    """
    Computes cross entropy between targets (encoded as one-hot vectors)
    and predictions. 
    Input: predictions (N, k) ndarray
           targets (N, k) ndarray        
    Returns: scalar
    """
    predictions = np.clip(predictions, epsilon, 1. - epsilon)
    ce = - np.mean(np.log(predictions) * targets) 
    return ce

以下代码将用于检查功能cross_entropy是否正确.

The following code will be used to check if the function cross_entropy are correct.

predictions = np.array([[0.25,0.25,0.25,0.25],
                        [0.01,0.01,0.01,0.96]])
targets = np.array([[0,0,0,1],
                  [0,0,0,1]])
ans = 0.71355817782  #Correct answer
x = cross_entropy(predictions, targets)
print(np.isclose(x,ans))

以上代码的输出为False，也就是说我定义函数cross_entropy的代码不正确.然后，打印cross_entropy(predictions, targets)的结果.它给出了0.178389544455，正确的结果应该是ans = 0.71355817782.有人可以帮我检查我的代码有什么问题吗?

The output of the above codes is False, that to say my codes for defining the function cross_entropy is not correct. Then I print the result of cross_entropy(predictions, targets). It gave 0.178389544455 and the correct result should be ans = 0.71355817782. Could anybody help me to check what is the problem with my codes?

推荐答案

您一点也不遥不可及，但请记住，您取的是N个和的平均值，其中N = 2(在这种情况下).因此您的代码可以显示为:

You're not that far off at all, but remember you are taking the average value of N sums, where N = 2 (in this case). So your code could read:

def cross_entropy(predictions, targets, epsilon=1e-12):
    """
    Computes cross entropy between targets (encoded as one-hot vectors)
    and predictions. 
    Input: predictions (N, k) ndarray
           targets (N, k) ndarray        
    Returns: scalar
    """
    predictions = np.clip(predictions, epsilon, 1. - epsilon)
    N = predictions.shape[0]
    ce = -np.sum(targets*np.log(predictions+1e-9))/N
    return ce

predictions = np.array([[0.25,0.25,0.25,0.25],
                        [0.01,0.01,0.01,0.96]])
targets = np.array([[0,0,0,1],
                   [0,0,0,1]])
ans = 0.71355817782  #Correct answer
x = cross_entropy(predictions, targets)
print(np.isclose(x,ans))

在这里，如果您坚持使用np.sum()，我想这会更清楚一些.另外，我在np.log()中添加了1e-9，以避免在计算中出现log(0)的可能性.希望这会有所帮助！

Here, I think it's a little clearer if you stick with np.sum(). Also, I added 1e-9 into the np.log() to avoid the possibility of having a log(0) in your computation. Hope this helps!

注意:根据@Peter的评论，如果您的epsilon值大于0，则1e-9的偏移量确实是多余的.

NOTE: As per @Peter's comment, the offset of 1e-9 is indeed redundant if your epsilon value is greater than 0.

这篇关于我的交叉熵函数的实现有什么问题?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！