PyTorch-如何在评估模式下停用辍学 [英] PyTorch - How to deactivate dropout in evaluation mode
问题描述
这是我定义的模型，它是一个具有2个完全连接层的简单lstm。
This is the model I defined it is a simple lstm with 2 fully connect layers.
import copy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
class mylstm(nn.Module):
def __init__(self,input_dim, output_dim, hidden_dim,linear_dim):
super(mylstm, self).__init__()
self.hidden_dim=hidden_dim
self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
self.linear1=nn.Linear(hidden_dim,linear_dim)
self.linear2=nn.Linear(linear_dim,output_dim)
def forward(self, input):
out,_=self.lstm(input)
out=nn.Dropout(p=0.3)(out)
out=self.linear1(out)
out=nn.Dropout(p=0.3)(out)
out=self.linear2(out)
return out
x_train 和 x_val 是形状为的float数据框（4478,30），而 y_ train 和 y_val 是形状为（4478,10）
x_train and x_val are float dataframe with shape (4478,30), while y_train and y_val are float df with shape (4478,10)
x_train.head()
Out[271]:
0 1 2 3 ... 26 27 28 29
0 1.6110 1.6100 1.6293 1.6370 ... 1.6870 1.6925 1.6950 1.6905
1 1.6100 1.6293 1.6370 1.6530 ... 1.6925 1.6950 1.6905 1.6960
2 1.6293 1.6370 1.6530 1.6537 ... 1.6950 1.6905 1.6960 1.6930
3 1.6370 1.6530 1.6537 1.6620 ... 1.6905 1.6960 1.6930 1.6955
4 1.6530 1.6537 1.6620 1.6568 ... 1.6960 1.6930 1.6955 1.7040
[5 rows x 30 columns]
x_train.shape
Out[272]: (4478, 30)
定义变量并做一次bp，我可以发现波动损失为1.4941
Define the varible and do one time bp, I can find out the vaildation loss is 1.4941
model=mylstm(30,10,200,100).double()
from torch import optim
optimizer=optim.RMSprop(model.parameters(), lr=0.001, alpha=0.9)
criterion=nn.L1Loss()
input_=torch.autograd.Variable(torch.from_numpy(np.array(x_train)))
target=torch.autograd.Variable(torch.from_numpy(np.array(y_train)))
input2_=torch.autograd.Variable(torch.from_numpy(np.array(x_val)))
target2=torch.autograd.Variable(torch.from_numpy(np.array(y_val)))
optimizer.zero_grad()
output=model(input_)
loss=criterion(output,target)
loss.backward()
optimizer.step()
moniter=criterion(model(input2_),target2)
moniter
Out[274]: tensor(1.4941, dtype=torch.float64, grad_fn=<L1LossBackward>)
但是由于辍学的随机性，我再次调用了正向函数，但得到了另一个数字
But I called forward function again I get a different number due to randomness of dropout
moniter=criterion(model(input2_),target2)
moniter
Out[275]: tensor(1.4943, dtype=torch.float64, grad_fn=<L1LossBackward>)
我该怎么做才能消除所有
what should I do that I can eliminate all the dropout in predicting phrase?
我试过 eval（）：
moniter=criterion(model.eval()(input2_),target2)
moniter
Out[282]: tensor(1.4942, dtype=torch.float64, grad_fn=<L1LossBackward>)
moniter=criterion(model.eval()(input2_),target2)
moniter
Out[283]: tensor(1.4945, dtype=torch.float64, grad_fn=<L1LossBackward>)
并传递附加参数p来控制辍学：
And pass an addtional parameter p to control dropout:
import copy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
class mylstm(nn.Module):
def __init__(self,input_dim, output_dim, hidden_dim,linear_dim,p):
super(mylstm, self).__init__()
self.hidden_dim=hidden_dim
self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
self.linear1=nn.Linear(hidden_dim,linear_dim)
self.linear2=nn.Linear(linear_dim,output_dim)
def forward(self, input,p):
out,_=self.lstm(input)
out=nn.Dropout(p=p)(out)
out=self.linear1(out)
out=nn.Dropout(p=p)(out)
out=self.linear2(out)
return out
model=mylstm(30,10,200,100,0.3).double()
output=model(input_)
loss=criterion(output,target)
loss.backward()
optimizer.step()
moniter=criterion(model(input2_,0),target2)
Traceback (most recent call last):
File "<ipython-input-286-e49b6fac918b>", line 1, in <module>
output=model(input_)
File "D:\Users\shan xu\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'p'
但它们都不起作用。
推荐答案
您必须在 __ init __ 中定义 nn.Dropout 层并进行分配使其响应您的模型以响应调用 eval（）。
You have to define your nn.Dropout layer in your __init__ and assign it to your model to be responsive for calling eval().
因此，像这样更改模型应适用于您：
So changing your model like this should work for you:
class mylstm(nn.Module):
def __init__(self,input_dim, output_dim, hidden_dim,linear_dim,p):
super(mylstm, self).__init__()
self.hidden_dim=hidden_dim
self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
self.linear1=nn.Linear(hidden_dim,linear_dim)
self.linear2=nn.Linear(linear_dim,output_dim)
# define dropout layer in __init__
self.drop_layer = nn.Dropout(p=p)
def forward(self, input):
out,_= self.lstm(input)
# apply model dropout, responsive to eval()
out= self.drop_layer(out)
out= self.linear1(out)
# apply model dropout, responsive to eval()
out= self.drop_layer(out)
out= self.linear2(out)
return out
如果您进行更改，则这样的退出会在您调用 eval时立即失效（ ）。
If you change it like this dropout will be inactive as soon as you call eval().
注意：如果以后要继续训练，则需要调用 train（）退出模型。
NOTE: If you want to continue training afterwards you need to call train() on your model to leave evaluation mode.
您还可以在此处找到用于评估模式的 eval（）的一个小型工作示例：
nn.Dropout与F.dropout pyTorch
You can also find a small working example for dropout with eval() for evaluation mode here:
nn.Dropout vs. F.dropout pyTorch
这篇关于PyTorch-如何在评估模式下停用辍学的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
