在Pytorch中使用Dropout:nn.Dropout与F.Dropout [英] Using Dropout in Pytorch: nn.Dropout vs. F.dropout

问题描述

通过使用pyTorch，有两种辍学方法 torch.nn.Dropout和torch.nn.functional.Dropout.

By using pyTorch there is two ways to dropout torch.nn.Dropout and torch.nn.functional.Dropout.

我很难看到它们之间的区别:

I struggle to see the difference between the use of them:

• 什么时候使用什么?
  
• 有什么不同吗?
  

切换它们时，我看不到任何性能差异.

I don't see any performance difference when I switched them around.

推荐答案

技术差异已在其他答案中显示.但是主要区别在于nn.Dropout是一个手电筒模块，具有一些便利:

The technical differences have already been shown in the other answer. However the main difference is that nn.Dropout is a torch Module itself which bears some convenience:

用于说明某些差异的简短示例:

A short example for illustration of some differences:

import torch
import torch.nn as nn

class Model1(nn.Module):
    # Model 1 using functional dropout
    def __init__(self, p=0.0):
        super().__init__()
        self.p = p

    def forward(self, inputs):
        return nn.functional.dropout(inputs, p=self.p, training=True)

class Model2(nn.Module):
    # Model 2 using dropout module
    def __init__(self, p=0.0):
        super().__init__()
        self.drop_layer = nn.Dropout(p=p)

    def forward(self, inputs):
        return self.drop_layer(inputs)
model1 = Model1(p=0.5) # functional dropout 
model2 = Model2(p=0.5) # dropout module

# creating inputs
inputs = torch.rand(10)
# forwarding inputs in train mode
print('Normal (train) model:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
print()

# switching to eval mode
model1.eval()
model2.eval()

# forwarding inputs in evaluation mode
print('Evaluation mode:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
# show model summary
print('Print summary:')
print(model1)
print(model2)

输出:

Normal (train) model:
Model 1 tensor([ 1.5040,  0.0000,  0.0000,  0.8563,  0.0000,  0.0000,  1.5951,
         0.0000,  0.0000,  0.0946])
Model 2 tensor([ 0.0000,  0.3713,  1.9303,  0.0000,  0.0000,  0.3574,  0.0000,
         1.1273,  1.5818,  0.0946])

Evaluation mode:
Model 1 tensor([ 0.0000,  0.3713,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
         0.0000,  0.0000,  0.0000])
Model 2 tensor([ 0.7520,  0.1857,  0.9651,  0.4281,  0.7883,  0.1787,  0.7975,
         0.5636,  0.7909,  0.0473])
Print summary:
Model1()
Model2(
  (drop_layer): Dropout(p=0.5)
)

那我应该使用哪个?

就辍学而言，两者完全相同，即使用法差异不大，也有一些原因使nn.Dropout优于nn.functional.dropout:

Both are completely equivalent in terms of applying dropout and even though the differences in usage are not that big, there are some reasons to favour the nn.Dropout over nn.functional.dropout:

Dropout被设计为仅在训练期间应用，因此在进行模型的预测或评估时，您希望关闭Dropout.

Dropout is designed to be only applied during training, so when doing predictions or evaluation of the model you want dropout to be turned off.

辍学模块nn.Dropout可以方便地处理此问题，并在模型进入评估模式后立即关闭辍学，而功能辍学并不在乎评估/预测模式.

The dropout module nn.Dropout conveniently handles this and shuts dropout off as soon as your model enters evaluation mode, while the functional dropout does not care about the evaluation / prediction mode.

即使您可以将功能设置为training=False以将其关闭，它仍然不是像nn.Dropout这样方便的解决方案.

Even though you can set functional dropout to training=False to turn it off, it is still not such a convenient solution like with nn.Dropout.

下降率也存储在模块中，因此您不必将其保存在额外的变量中.在较大的网络中，您可能希望创建具有不同丢包率的不同丢包层-在这里nn.Dropout可能会提高可读性，并且在多次使用这些层时也可以带来一些便利.

Also the drop rate is stored in the module, so you don't have to save it in an extra variable. In larger networks you might want to create different dropout layers with different drop rates - here nn.Dropout may increase readability and can bear also some convenience when using the layers multiple times.

最后，分配给您模型的所有模块都将在您的模型中注册.因此，您的模型类会跟踪它们，这就是为什么您可以通过调用eval()来关闭Dropout模块的原因.使用功能性辍学时，您的模型不会意识到这一点，因此它不会出现在任何摘要中.

Finally, all modules which are assigned to your model are registered in your model. So you model class keeps track of them, that is why you can just turn off the dropout module by calling eval(). When using the functional dropout your model is not aware of it, thus it won't appear in any summary.

这篇关于在Pytorch中使用Dropout:nn.Dropout与F.Dropout的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！