在 Pytorch 中使用 Dropout:nn.Dropout 与 F.dropout [英] Using Dropout in Pytorch: nn.Dropout vs. F.dropout

问题描述

通过使用 pyTorch 有两种方法可以退出torch.nn.Dropout 和 torch.nn.functional.Dropout.

By using pyTorch there is two ways to dropout torch.nn.Dropout and torch.nn.functional.Dropout.

我很难看出它们的使用之间的区别:

I struggle to see the difference between the use of them:

• 何时使用什么?
  
• 这有什么不同吗?
  

当我切换它们时，我没有看到任何性能差异.

I don't see any performance difference when I switched them around.

推荐答案

技术差异已在另一个答案中显示.然而，主要区别在于 nn.Dropout 是一个火炬模块本身，它具有一些便利性:

The technical differences have already been shown in the other answer. However the main difference is that nn.Dropout is a torch Module itself which bears some convenience:

说明一些差异的简短示例:

A short example for illustration of some differences:

import torch
import torch.nn as nn

class Model1(nn.Module):
    # Model 1 using functional dropout
    def __init__(self, p=0.0):
        super().__init__()
        self.p = p

    def forward(self, inputs):
        return nn.functional.dropout(inputs, p=self.p, training=True)

class Model2(nn.Module):
    # Model 2 using dropout module
    def __init__(self, p=0.0):
        super().__init__()
        self.drop_layer = nn.Dropout(p=p)

    def forward(self, inputs):
        return self.drop_layer(inputs)
model1 = Model1(p=0.5) # functional dropout 
model2 = Model2(p=0.5) # dropout module

# creating inputs
inputs = torch.rand(10)
# forwarding inputs in train mode
print('Normal (train) model:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
print()

# switching to eval mode
model1.eval()
model2.eval()

# forwarding inputs in evaluation mode
print('Evaluation mode:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
# show model summary
print('Print summary:')
print(model1)
print(model2)

输出:

Normal (train) model:
Model 1 tensor([ 1.5040,  0.0000,  0.0000,  0.8563,  0.0000,  0.0000,  1.5951,
         0.0000,  0.0000,  0.0946])
Model 2 tensor([ 0.0000,  0.3713,  1.9303,  0.0000,  0.0000,  0.3574,  0.0000,
         1.1273,  1.5818,  0.0946])

Evaluation mode:
Model 1 tensor([ 0.0000,  0.3713,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
         0.0000,  0.0000,  0.0000])
Model 2 tensor([ 0.7520,  0.1857,  0.9651,  0.4281,  0.7883,  0.1787,  0.7975,
         0.5636,  0.7909,  0.0473])
Print summary:
Model1()
Model2(
  (drop_layer): Dropout(p=0.5)
)

那我应该用哪个?

两者在应用 dropout 方面完全相同，即使使用差异不大，也有一些理由支持 nn.Dropout 而不是 nn.functional.dropout:

Both are completely equivalent in terms of applying dropout and even though the differences in usage are not that big, there are some reasons to favour the nn.Dropout over nn.functional.dropout:

Dropout 旨在仅在训练期间应用，因此在对模型进行预测或评估时，您希望关闭 dropout.

Dropout is designed to be only applied during training, so when doing predictions or evaluation of the model you want dropout to be turned off.

dropout 模块 nn.Dropout 可以方便地处理这个问题，并在您的模型进入评估模式后立即关闭 dropout，而功能 dropout 不关心评估/预测模式.

The dropout module nn.Dropout conveniently handles this and shuts dropout off as soon as your model enters evaluation mode, while the functional dropout does not care about the evaluation / prediction mode.

即使您可以将功能性dropout设置为training=False来关闭它，但它仍然不是像nn.Dropout<那样方便的解决方案/代码>.

Even though you can set functional dropout to training=False to turn it off, it is still not such a convenient solution like with nn.Dropout.

此外，掉落率也存储在模块中，因此您不必将其保存在额外的变量中.在较大的网络中，您可能希望创建具有不同丢弃率的不同丢弃层 - 这里 nn.Dropout 可以提高可读性，并且在多次使用这些层时也可以带来一些便利.

Also the drop rate is stored in the module, so you don't have to save it in an extra variable. In larger networks you might want to create different dropout layers with different drop rates - here nn.Dropout may increase readability and can bear also some convenience when using the layers multiple times.

最后，分配给您的模型的所有模块都在您的模型中注册.所以你的模型类会跟踪它们，这就是为什么你可以通过调用 eval() 来关闭 dropout 模块.当使用函数 dropout 时，您的模型不会意识到它，因此它不会出现在任何摘要中.

Finally, all modules which are assigned to your model are registered in your model. So you model class keeps track of them, that is why you can just turn off the dropout module by calling eval(). When using the functional dropout your model is not aware of it, thus it won't appear in any summary.

这篇关于在 Pytorch 中使用 Dropout:nn.Dropout 与 F.dropout的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！