在 PyTorch 中，默认情况下如何初始化层权重和偏差? [英] In PyTorch how are layer weights and biases initialized by default?

问题描述

我想知道默认情况下层权重和偏差是如何初始化的?例如.如果我创建线性层torch.nn.Linear(5,100)默认情况下，该层的权重和偏差如何初始化?

I was wondering how are layer weights and biases initialized by default? E.g. if I create the linear layer torch.nn.Linear(5,100) How are weights and biases for this layer initialized by default?

推荐答案

PyTorch 1.0

大多数层使用Kaiming Uniform 方法初始化.示例层包括 Linear、Conv2d、RNN 等.如果您正在使用其他层，您应该在 此文档.如果它说使用 U(...) 初始化权重，那么它的 Kaiming Uniform 方法.偏差使用 LeCunn init 初始化，即 uniform(-std, std) 其中标准差 std 是 1/sqrt(fan_in) (代码).

Most layers are initialized using Kaiming Uniform method. Example layers include Linear, Conv2d, RNN etc. If you are using other layers, you should look up that layer on this doc. If it says weights are initialized using U(...) then its Kaiming Uniform method. Bias is initialized using LeCunn init, i.e., uniform(-std, std) where standard deviation std is 1/sqrt(fan_in) (code).

PyTorch 0.4.1、0.3.1

权重和偏差使用 LeCunn init 初始化(参见sec 4.6)用于转换层(代码:0.3.1，0.4.1).

Weights and biases are initialized using LeCunn init (see sec 4.6) for conv layers (code: 0.3.1, 0.4.1).

如果您想覆盖默认初始化，请查看此答案.

If you want to override default initialization then see this answer.

这篇关于在 PyTorch 中，默认情况下如何初始化层权重和偏差?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！