PyTorch-nn的形状线性权重 [英] PyTorch - shape of nn.Linear weights

问题描述

昨天我遇到了

Yesterday I came across this question and for the first time noticed that the weights of the linear layer nn.Linear need to be transposed before applying matmul.

代码来应用权重:

Code for applying the weights:

output = input.matmul(weight.t())

这是什么原因?

为什么权重从一开始就不处于转置形状，因此在应用图层之前不必每次都进行移置?

Why are the weights not in the transposed shape just from the beginning, so they don't need to be transposed every time before applying the layer?

推荐答案

我在这里找到了答案: nn.Linear#2159中的有效前向通过

I found an answer here: Efficient forward pass in nn.Linear #2159

似乎这背后没有任何真正的道理.但是，转置操作似乎并不会减慢计算速度.

It seems like there is no real reasoning behind this. However the transpose operation doesn't seem to be slowing down the computation.

根据上述问题，在前进过程中，转置操作在计算方面几乎(免费).在向后传递期间，省去转置操作实际上会使当前实现的计算效率降低.

According to the issue mentioned above, during the forward pass the transpose operation is (almost) free in terms of computation. While during the backward pass leaving out the transpose operation would actually make computation less efficient with the current implementation.

该期的最后一篇文章很好地总结了这一点:

The last post in that issue sums it up quite nicely:

这是历史的重量布局，对其进行更改是向后不兼容的. 除非在速度或便利性方面有很大的好处，否则我们 不会破坏用户空间.

It's historical weight layout, changing it is backward-incompatible. Unless there is some BIG benefit in terms of speed or convenience, we wont break userland.

https://github.com/pytorch/pytorch/issues/2159#issuecomment-390068272

这篇关于PyTorch-nn的形状线性权重的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！