如何为 GradientDescentOptimizer 设置自适应学习率? [英] How to set adaptive learning rate for GradientDescentOptimizer?
问题描述
我正在使用 TensorFlow 训练神经网络.这就是我初始化 GradientDescentOptimizer
的方式:
I am using TensorFlow to train a neural network. This is how I am initializing the GradientDescentOptimizer
:
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
mse = tf.reduce_mean(tf.square(out - out_))
train_step = tf.train.GradientDescentOptimizer(0.3).minimize(mse)
这里的问题是我不知道如何为学习率或衰减值设置更新规则.
The thing here is that I don't know how to set an update rule for the learning rate or a decay value for that.
如何在此处使用自适应学习率?
How can I use an adaptive learning rate here?
推荐答案
首先,tf.train.GradientDescentOptimizer
旨在对所有步骤中的所有变量使用恒定的学习率.TensorFlow 还提供开箱即用的自适应优化器,包括 tf.train.AdagradOptimizer代码>
和 tf.train.AdamOptimizer
,这些可以用作替代品.
First of all, tf.train.GradientDescentOptimizer
is designed to use a constant learning rate for all variables in all steps. TensorFlow also provides out-of-the-box adaptive optimizers including the tf.train.AdagradOptimizer
and the tf.train.AdamOptimizer
, and these can be used as drop-in replacements.
然而,如果你想用其他普通的梯度下降来控制学习率,你可以利用 learning_rate
参数到 tf.train.GradientDescentOptimizer
构造函数 可以是 Tensor
对象.这允许您为每个步骤中的学习率计算不同的值,例如:
However, if you want to control the learning rate with otherwise-vanilla gradient descent, you can take advantage of the fact that the learning_rate
argument to the tf.train.GradientDescentOptimizer
constructor can be a Tensor
object. This allows you to compute a different value for the learning rate in each step, for example:
learning_rate = tf.placeholder(tf.float32, shape=[])
# ...
train_step = tf.train.GradientDescentOptimizer(
learning_rate=learning_rate).minimize(mse)
sess = tf.Session()
# Feed different values for learning rate to each training step.
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.01})
sess.run(train_step, feed_dict={learning_rate: 0.01})
或者,您可以创建一个标量 tf.Variable
来保存学习率,并在每次想要更改学习率时分配它.
Alternatively, you could create a scalar tf.Variable
that holds the learning rate, and assign it each time you want to change the learning rate.
这篇关于如何为 GradientDescentOptimizer 设置自适应学习率?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!