如何根据 PyTorch 中的概率计算交叉熵? [英] How do I calculate cross-entropy from probabilities in PyTorch?

问题描述

默认情况下，PyTorch 的 cross_entropy 将 logits(模型的原始输出)作为输入.我知道 CrossEntropyLoss 将 LogSoftmax (log(softmax(x))) 和 NLLLoss(负对数似然损失)组合在一个类中.所以，我想我可以使用 NLLLoss 从概率中获得交叉熵损失，如下所示:

<块引用>

真实标签:[1, 0, 1]
概率:[0.1, 0.9], [0.9, 0.1], [0.2, 0.8]

其中，y_i,j 表示真实值，即如果样本 i 属于类 j 则为 1，否则为 0.和 p_i,j 表示您的模型预测的概率属于类 j 的样本 i.

如果我手工计算，结果是:

<预><代码>>>>-(math.log(0.9) + math.log(0.9) + math.log(0.8))0.4338

使用 PyTorch:

<预><代码>>>>标签 = torch.tensor([1, 0, 1], dtype=torch.long)>>>probs = torch.tensor([[0.1, 0.9], [0.9, 0.1], [0.2, 0.8]], dtype=torch.float)>>>F.nll_loss(torch.log(probs), 标签)张量(0.1446)

我做错了什么?为什么答案不同?

解决方案

PyTorch 中的所有损失函数都有一个归约参数.正如您从 文档 默认减少参数是均值"，它将总和除以批次中的元素数.要根据需要获得求和行为 (0.4338)，您应该提供如下的归约参数:

F.nll_loss(torch.log(probs), labels,reduction='sum')

By default, PyTorch's cross_entropy takes logits (the raw outputs from the model) as the input. I know that CrossEntropyLoss combines LogSoftmax (log(softmax(x))) and NLLLoss (negative log likelihood loss) in one single class. So, I think I can use NLLLoss to get cross-entropy loss from probabilities as follows:

true labels: [1, 0, 1]
probabilites: [0.1, 0.9], [0.9, 0.1], [0.2, 0.8]

where, y_i,j denotes the true value i.e. 1 if sample i belongs to class j and 0 otherwise. and p_i,j denotes the probability predicted by your model of sample i belonging to class j.

If I calculate by hand, it turns out to be:

>>> -(math.log(0.9) + math.log(0.9) + math.log(0.8))
0.4338

Using PyTorch:

>>> labels = torch.tensor([1, 0, 1], dtype=torch.long)
>>> probs = torch.tensor([[0.1, 0.9], [0.9, 0.1], [0.2, 0.8]], dtype=torch.float)
>>> F.nll_loss(torch.log(probs), labels)
tensor(0.1446)

What am I doing wrong? Why is the answer different?

解决方案

There is a reduction parameter for all loss functions in the PyTorch. As you can see from the documentation default reduction parameter is 'mean' which divides the sum with number of elements in the batch. To get a summation behavior (0.4338) as you want, you should give the reduction parameter as following:

F.nll_loss(torch.log(probs), labels,reduction='sum')

这篇关于如何根据 PyTorch 中的概率计算交叉熵?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！