PyTorch 自定义损失函数 [英] PyTorch custom loss function
问题描述
自定义损失函数应该如何实现?使用以下代码导致错误:
导入火炬将 torch.nn 导入为 nn进口火炬视觉导入 torchvision.transforms 作为转换将 numpy 导入为 np导入 matplotlib.pyplot 作为 plt将 torch.utils.data 导入为 data_utils将 torch.nn 导入为 nn导入 torch.nn.functional 作为 Fnum_epochs = 20x1 = np.array([0,0])x2 = np.array([0,1])x3 = np.array([1,0])x4 = np.array([1,1])num_epochs = 200类 cus2(torch.nn.Module):def __init__(self):super(cus2,self).__init__()def forward(自我，输出，标签):# 重塑标签以给出长度为 batch_size*seq_len 的平面向量标签 = 标签.view(-1)# 屏蔽PAD"令牌掩码 = (标签 >= 0).float()#token的数量是mask中元素的总和num_tokens = int(torch.sum(mask).data[0])# 选择标签对应的值并乘以掩码输出=输出[范围(输出.形状[0])，标签]*掩码# 所有非PAD"令牌的交叉熵损失返回 -torch.sum(outputs)/num_tokensx = torch.tensor([x1,x2,x3,x4]).float()y = torch.tensor([0,1,1,0]).long()train = data_utils.TensorDataset(x,y)train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)设备 = 'cpu'输入大小 = 2隐藏尺寸 = 100num_classes = 2学习率 = .0001类神经网络(nn.Module):def __init__(self, input_size, hidden_size, num_classes) :super(NeuralNet, self).__init__()self.fc1 = nn.Linear(input_size , hidden_size)self.relu = nn.ReLU()self.fc2 = nn.Linear(hidden_size , num_classes)def forward(self, x) :out = self.fc1(x)out = self.relu(out)out = self.fc2(out)回来对于范围内的 i (0 , 1) :模型 = NeuralNet(input_size, hidden_size, num_classes).to(device)标准 = nn.CrossEntropyLoss()# 标准 = Regress_Loss()# 标准 = cus2()优化器 = torch.optim.Adam(model.parameters(), lr=learning_rate)total_step = len(train_loader)对于范围内的纪元(num_epochs):对于 enumerate(train_loader) 中的 i,(images , labels) :图像 = images.reshape(-1 , 2).to(device)标签 = 标签.to(设备)输出 = 模型(图像)损失 = 标准(输出，标签)optimizer.zero_grad()损失.向后()优化器.step()# 打印(损失)输出 = 模型(x)打印(输出.数据.最大(1)[1])对训练数据做出完美的预测:
张量([0, 1, 1, 0])使用来自
在上面的代码中实现为 cus2
取消注释代码 #criteria = cus2() 使用这个损失函数返回:
张量([0, 0, 0, 0])也会返回警告:
<块引用>用户警告:0 维张量的无效索引.这将是一个错误PyTorch 0.5.使用 tensor.item() 将 0-dim 张量转换为 Python号码
我没有正确实现自定义损失函数?
您的损失函数在编程上是正确的，除了以下内容:
#token的个数是mask中元素的总和num_tokens = int(torch.sum(mask).data[0])当你执行 torch.sum 时，它返回一个 0 维张量，因此警告它不能被索引.要解决此问题，请按照建议执行 int(torch.sum(mask).item()) 或 int(torch.sum(mask)) 也可以.
现在，您是否尝试使用自定义损失来模拟 CE 损失?如果是，那么您缺少 log_softmax
要修复在语句 4 之前添加 outputs = torch.nn.functional.log_softmax(outputs, dim=1).请注意，如果您附加了教程，log_softmax 已经在前向调用中完成.你也可以那样做.
另外，我注意到学习速度很慢，即使有 CE 损失，结果也不一致.在自定义和 CE 损失的情况下，将学习率增加到 1e-3 对我来说效果很好.
How should a custom loss function be implemented ? Using below code is causing error :
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F
num_epochs = 20
x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])
num_epochs = 200
class cus2(torch.nn.Module):
def __init__(self):
super(cus2,self).__init__()
def forward(self, outputs, labels):
# reshape labels to give a flat vector of length batch_size*seq_len
labels = labels.view(-1)
# mask out 'PAD' tokens
mask = (labels >= 0).float()
# the number of tokens is the sum of elements in mask
num_tokens = int(torch.sum(mask).data[0])
# pick the values corresponding to labels and multiply by mask
outputs = outputs[range(outputs.shape[0]), labels]*mask
# cross entropy loss for all non 'PAD' tokens
return -torch.sum(outputs)/num_tokens
x = torch.tensor([x1,x2,x3,x4]).float()
y = torch.tensor([0,1,1,0]).long()
train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)
device = 'cpu'
input_size = 2
hidden_size = 100
num_classes = 2
learning_rate = .0001
class NeuralNet(nn.Module) :
def __init__(self, input_size, hidden_size, num_classes) :
super(NeuralNet, self).__init__()
self.fc1 = nn.Linear(input_size , hidden_size)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(hidden_size , num_classes)
def forward(self, x) :
out = self.fc1(x)
out = self.relu(out)
out = self.fc2(out)
return out
for i in range(0 , 1) :
model = NeuralNet(input_size, hidden_size, num_classes).to(device)
criterion = nn.CrossEntropyLoss()
# criterion = Regress_Loss()
# criterion = cus2()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
total_step = len(train_loader)
for epoch in range(num_epochs) :
for i,(images , labels) in enumerate(train_loader) :
images = images.reshape(-1 , 2).to(device)
labels = labels.to(device)
outputs = model(images)
loss = criterion(outputs , labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# print(loss)
outputs = model(x)
print(outputs.data.max(1)[1])
makes perfect predictions on training data :
tensor([0, 1, 1, 0])
Using a custom loss function from here:
is implemented in above code as cus2
Un-commenting code # criterion = cus2() to use this loss function returns :
tensor([0, 0, 0, 0])
A warning is also returned :
UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
I've not implemented the custom loss function correctly ?
Your loss function is programmatically correct except for below:
# the number of tokens is the sum of elements in mask
num_tokens = int(torch.sum(mask).data[0])
When you do torch.sum it returns a 0-dimensional tensor and hence the warning that it can't be indexed. To fix this do int(torch.sum(mask).item()) as suggested or int(torch.sum(mask)) will work too.
Now, are you trying to emulate the CE loss using the custom loss? If yes, then you are missing the log_softmax
To fix that add outputs = torch.nn.functional.log_softmax(outputs, dim=1) before statement 4. Note that in case of tutorial that you have attached, log_softmax is already done in the forward call. You can do that too.
Also, I noticed that the learning rate is slow and even with CE loss, results are not consistent. Increasing the learning rate to 1e-3 works well for me in case of custom as well as CE loss.
这篇关于PyTorch 自定义损失函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
