softmax_cross_entropy_with_logits 和 loss.log_loss 有什么区别? [英] what's the difference between softmax_cross_entropy_with_logits and losses.log_loss?

问题描述

tf.nn.softmax_cross_entropy_with_logits 和 tf.losses.log_loss 之间的主要区别是什么?两种方法都接受 1-hot 标签和 logits 来计算分类任务的交叉熵损失.

whats the primary difference between tf.nn.softmax_cross_entropy_with_logits and tf.losses.log_loss? both methods accept 1-hot labels and logits to calculate cross entropy loss for classification tasks.

推荐答案

这些方法在理论上没有太大区别，但在实现上有很多不同:

Those methods are not so different in theory, however have number of differences in implementation:

1) tf.nn.softmax_cross_entropy_with_logits 设计用于单类标签，而 tf.losses.log_loss 可用于多类分类.如果您提供多类标签，tf.nn.softmax_cross_entropy_with_logits 不会抛出错误，但是您的梯度将无法正确计算并且训练很可能会失败.

1) tf.nn.softmax_cross_entropy_with_logitsis designed for single-class labels, while tf.losses.log_losscan be used for multi-class classification. tf.nn.softmax_cross_entropy_with_logits won't throw an error if you feed multi-class labels, however your gradients won't be calculated correctly and training most probably will fail.

来自官方文档:

注意:虽然这些类是互斥的，但它们的概率不一定是.所需要的只是每一行标签都是一个有效的概率分布.如果不是，梯度的计算将是错误的.

NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

2) tf.nn.softmax_cross_entropy_with_logits 首先在您的预测之上计算(从名称中可以看出)soft-max 函数，而 log_loss 不这样做.

2) tf.nn.softmax_cross_entropy_with_logits calculates (as it's seen from the name) soft-max function on top of your predictions first, while log_loss doesn't do this.

3) tf.losses.log_loss 在某种意义上具有更广泛的功能，您可以对损失函数的每个元素进行加权，也可以指定 epsilon，即用于计算，以避免 log(0) 值.

3) tf.losses.log_loss has a little wider functionality in a sense that you can weight each element of the loss function or you can specify epsilon, which is used in calculations, to avoid log(0) value.

4) 最后，tf.nn.softmax_cross_entropy_with_logits 返回批次中每个条目的损失，而 tf.losses.log_loss 返回减少的(默认情况下所有样本的总和) 可以直接在优化器中使用的值.

4) Finally, tf.nn.softmax_cross_entropy_with_logits returns loss for every entry in the batch, while tf.losses.log_loss returns reduced (sum over all samples by default) value which can be directly used in optimizer.

UPD:另一个区别是计算损失的方式，对数损失考虑了负类(向量中有 0 的那些).很快，交叉熵损失迫使网络为正确的类产生最大输入，而不关心负类.对数损失同时进行，它迫使正确的类具有更大的值而负值更小.数学表达式如下:

UPD: Another difference is the way the calculate the loss, Logarithmic loss takes into account negative classes (those where you have 0s in the vector). Shortly, cross-enthropy loss forces network to produce maximum input for the correct class and does not care about negative classes. Logarithmic loss does both at the same time, it forces correct classes to have larger values and negative lesser. In mathematic expression it looks as following:

交叉熵损失:

对数损失:

其中 i 是对应的类.

Where i is the corresponding class.

例如，如果您有 labels=[1,0] 和 predictions_with_softmax = [0.7,0.3]，那么:

So for example, if you have labels=[1,0] and predictions_with_softmax = [0.7,0.3], then:

1) 交叉熵损失:-(1 * log(0.7) + 0 * log(0.3)) = 0.3567

1) Cross-Enthropy Loss: -(1 * log(0.7) + 0 * log(0.3)) = 0.3567

2) 对数损失: - (1*log(0.7) + (1-1) * log(1 - 0.7) +0*log(0.3) + (1-0) log (1-0.3)) =- (log(0.7) + log (0.7)) = 0.7133

2) Logarithmic Loss: - (1*log(0.7) + (1-1) * log(1 - 0.7) +0*log(0.3) + (1-0) log (1- 0.3)) = - (log(0.7) + log (0.7)) = 0.7133

然后，如果您使用 tf.losses.log_loss 的默认值，则需要将 log_loss 输出除以非零元素的数量(这里是 2).所以最后:tf.nn.log_loss = 0.7133/2 = 0.3566

And then if you use default value for tf.losses.log_loss you then need to divide the log_loss output by the number of non-zero elements (here it's 2). So finally: tf.nn.log_loss = 0.7133 / 2 = 0.3566

在这种情况下，我们得到了相等的输出，但情况并非总是如此

In this case we got equal outputs, however it is not always the case

这篇关于softmax_cross_entropy_with_logits 和 loss.log_loss 有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！