如何打印“实际"标签pytorch中Adadelta的学习率 [英] How to print the "actual" learning rate in Adadelta in pytorch

问题描述

简而言之:

在pytorch中使用adadelta优化器时，我无法绘制lr/时期曲线，因为 optimizer.param_groups [0] ['lr'] 始终返回相同的值.

I can't draw lr/epoch curve when using adadelta optimizer in pytorch because optimizer.param_groups[0]['lr'] always return the same value.

详细信息:

Adadelta可以仅使用一阶信息就可以动态适应时间变化，并且具有超出香草随机梯度下降的最小计算开销[1].

Adadelta can dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent [1].

在pytorch中，Adadelta的源代码在这里 https://pytorch.org/docs/stable/_modules/torch/optim/adadelta.html#Adadelta

In pytorch, the source code of Adadelta is here https://pytorch.org/docs/stable/_modules/torch/optim/adadelta.html#Adadelta

由于不需要手动调整学习率，据我所知，在声明优化器后，我们不必设置任何时间表

Since it requires no manual tuning of learning rate, in my knowledge, we don't have to set any schedular after declare the optimizer

self.optimizer = torch.optim.Adadelta(self.model.parameters()，lr = 1)

检查学习率的方法是

current_lr = self.optimizer.param_groups [0] ['lr']

问题在于它总是返回1(初始lr).

有人可以告诉我如何获得真正的学习率，以便可以画出lr/epch曲线吗?

Could anyone tell me how can I get the true learning rate so that can I draw a lr/epch curve?

[1] https://arxiv.org/pdf/1212.5701.pdf

推荐答案

检查: self.optimizer.state .这使用lr进行了优化，并在优化过程中使用.

Check: self.optimizer.state. This is optimized with the lr and used in optimization process.

在文档中，lr只是:

lr(浮点型，可选):在增量delta之前标度的系数已应用设置为参数(默认值:1.0)

lr (float, optional): coefficient that scale delta before it is applied to the parameters (default: 1.0)

https://pytorch.org/docs/stable/_modules/torch/optim/adadelta.html

您可能会在self.optimizer.state值中找到acc_delta值，但需要仔细阅读此词典中包含的字典:

Edited: you may find acc_delta values in self.optimizer.state values but you need to go through dictionaries contained by this dictionary:

dict_with_acc_delta = [self.optimizer.state[i] for i in self.optimizer.state.keys() if "acc_delta" in self.optimizer.state[i].keys()]
acc_deltas = [i["acc_delta"] for i in dict_with_acc_delta]

我有八层，acc_deltas列表中的元素形状如下

I have eight layers and shapes of elements in the acc_deltas list are following

[torch.Size([25088]),
 torch.Size([25088]),
 torch.Size([4096, 25088]),
 torch.Size([4096]),
 torch.Size([1024, 4096]),
 torch.Size([1024]),
 torch.Size([102, 1024]),
 torch.Size([102])]

这篇关于如何打印“实际"标签pytorch中Adadelta的学习率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！