PyTorch:学习率调度程序 [英] PyTorch: Learning rate scheduler

问题描述

如何将学习率调度程序与以下优化程序一起使用?

How do I use a learning rate scheduler with the following optimizer?

optimizer = torch.optim.Adam(optim_params,betas=(args.momentum, args.beta), weight_decay=args.weight_decay)

我编写了以下调度程序:

I have written the following scheduler:

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.9)

我不清楚是否应该执行调度程序或优化程序.我应该采取哪个顺序来执行以下操作?

I am not clear if I should step the scheduler or the optimizer. Which order should I take to perform the following?

optimizer.zero_grad()
scheduler.step()
optimizer.step()

推荐答案

由于1.3行为已更改，请参见此问题尤其重要.

Since 1.3 the behaviour was changed, see releases and this issue especially.

在此版本之前，您应该在optimizer之前的step调度程序，这在IMO中是不合理的.来回有一些冲突(实际上，它破坏了向后兼容性，而IMO则因为这么小的不便而破坏它不是一个好主意)，但是目前您应该在optimizer之后执行scheduler.

Before this version, you should step scheduler before optimizer, which IMO wasn't reasonable. There was some back and forth (actually it breaks backward compatibility and IMO it's not a good idea to break it for such a minor inconvenience), but currently you should step scheduler after optimizer.

optimizer.zero_grad()
optimizer.step()
scheduler.step()

这篇关于PyTorch:学习率调度程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！