为什么我们需要在PyTorch中调用zero_grad()? [英] Why do we need to call zero_grad() in PyTorch?

问题描述

在训练过程中需要调用方法zero_grad().但是文档不是很有帮助

The method zero_grad() needs to be called during training. But the documentation is not very helpful

|  zero_grad(self)
|      Sets gradients of all model parameters to zero.

我们为什么需要调用此方法?

Why do we need to call this method?

推荐答案

在 PyTorch 中，我们需要设置在开始反向传播之前将梯度降为零，因为PyTorch 会在随后的向后传递中积累梯度.在训练RNN时这很方便.因此，默认操作是在渐变上累积(即求和)每次loss.backward()调用.

In PyTorch, we need to set the gradients to zero before starting to do backpropragation because PyTorch accumulates the gradients on subsequent backward passes. This is convenient while training RNNs. So, the default action is to accumulate (i.e. sum) the gradients on every loss.backward() call.

因此，在开始训练循环时，理想情况下，您应该

Because of this, when you start your training loop, ideally you should zero out the gradients so that you do the parameter update correctly. Else the gradient would point in some other direction than the intended direction towards the minimum (or maximum, in case of maximization objectives).

这是一个简单的例子:

import torch
from torch.autograd import Variable
import torch.optim as optim

def linear_model(x, W, b):
    return torch.matmul(x, W) + b

data, targets = ...

W = Variable(torch.randn(4, 3), requires_grad=True)
b = Variable(torch.randn(3), requires_grad=True)

optimizer = optim.Adam([W, b])

for sample, target in zip(data, targets):
    # clear out the gradients of all Variables 
    # in this optimizer (i.e. W, b)
    optimizer.zero_grad()
    output = linear_model(sample, W, b)
    loss = (output - target) ** 2
    loss.backward()
    optimizer.step()

---

或者，如果您要进行香草梯度下降，则:

---

Alternatively, if you're doing a vanilla gradient descent, then:

W = Variable(torch.randn(4, 3), requires_grad=True)
b = Variable(torch.randn(3), requires_grad=True)

for sample, target in zip(data, targets):
    # clear out the gradients of Variables 
    # (i.e. W, b)
    W.grad.data.zero_()
    b.grad.data.zero_()

    output = linear_model(sample, W, b)
    loss = (output - target) ** 2
    loss.backward()

    W -= learning_rate * W.grad.data
    b -= learning_rate * b.grad.data

---

注意:当

---

Note: The accumulation (i.e. sum) of gradients happen when .backward() is called on the loss tensor.

这篇关于为什么我们需要在PyTorch中调用zero_grad()?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！