PyTorch中的交叉熵 [英] Cross Entropy in PyTorch

问题描述

我对PyTorch中的交叉熵损失感到困惑.

I'm a bit confused by the cross entropy loss in PyTorch.

考虑此示例:

import torch
import torch.nn as nn
from torch.autograd import Variable

output = Variable(torch.FloatTensor([0,0,0,1])).view(1, -1)
target = Variable(torch.LongTensor([3]))

criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

我希望损失为0.但是我得到了:

I would expect the loss to be 0. But I get:

Variable containing:
 0.7437
[torch.FloatTensor of size 1]

据我所知，交叉熵可以这样计算:

As far as I know cross entropy can be calculated like this:

但是不应该是1 * log(1)= 0的结果吗?

But shouldn't be the result then 1*log(1) = 0 ?

我尝试了诸如热编码的不同输入，但这根本不起作用，因此损失函数的输入形状似乎还可以.

I tried different inputs like one-hot encodings, but this doesn't work at all, so it seems the input shape of the loss function is okay.

如果有人可以帮助我并告诉我我的错误在哪里，我将非常感激.

I would be really grateful if someone could help me out and tell me where my mistake is.

提前谢谢！

推荐答案

在您的示例中，您将输出 [0,0,0,1] 视为cross的数学定义所需的概率熵.但是PyTorch将它们视为输出，不需要求和为 1 ，并且需要首先转换为使用softmax函数的概率.

In your example you are treating output [0,0,0,1] as probabilities as required by the mathematical definition of cross entropy. But PyTorch treats them as outputs, that don’t need to sum to 1, and need to be first converted into probabilities for which it uses the softmax function.

因此 H(p，q) 变为:
H(p，softmax(output))

So H(p,q) becomes:
H(p, softmax(output))

将输出 [0,0,0,1] 转换为概率:
softmax([0,0,0,1])= [0.1749,0.1749,0.1749,0.4754]

Translating the output [0,0,0,1] into probabilities:
softmax([0,0,0,1])= [0.1749,0.1749,0.1749,0.4754]

其中:
-log(0.4754)= 0.7437

whence:
-log(0.4754) = 0.7437

这篇关于PyTorch中的交叉熵的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！