Tensorflow:使用 Adam 优化器 [英] Tensorflow: Using Adam optimizer

问题描述

我正在 tensorflow 中试验一些简单的模型，包括一个看起来与第一个 MNIST 非常相似的模型ML 初学者示例，但维度稍大.我可以毫无问题地使用梯度下降优化器，获得足够好的收敛性.当我尝试使用 ADAM 优化器时，出现如下错误:

I am experimenting with some simple models in tensorflow, including one that looks very similar to the first MNIST for ML Beginners example, but with a somewhat larger dimensionality. I am able to use the gradient descent optimizer with no problems, getting good enough convergence. When I try to use the ADAM optimizer, I get errors like this:

tensorflow.python.framework.errors.FailedPreconditionError: Attempting to use uninitialized value Variable_21/Adam
     [[Node: Adam_2/update_Variable_21/ApplyAdam = ApplyAdam[T=DT_FLOAT, use_locking=false, _device="/job:localhost/replica:0/task:0/cpu:0"](Variable_21, Variable_21/Adam, Variable_21/Adam_1, beta1_power_2, beta2_power_2, Adam_2/learning_rate, Adam_2/beta1, Adam_2/beta2, Adam_2/epsilon, gradients_11/add_10_grad/tuple/control_dependency_1)]]

抱怨未初始化的特定变量的变化取决于运行.这个错误是什么意思?它表明什么是错误的?无论我使用什么学习率，它似乎都会发生.

where the specific variable that complains about being uninitialized changes depending on the run. What does this error mean? And what does it suggest is wrong? It seems to occur regardless of the learning rate I use.

推荐答案

AdamOptimizer 类创建了额外的变量，称为槽"，以保存m"和v"累加器的值.

The AdamOptimizer class creates additional variables, called "slots", to hold values for the "m" and "v" accumulators.

如果您好奇，请在此处查看源代码，它实际上非常易读:https://github.com/tensorflow/tensorflow/blob/master/张量流/python/training/adam.py#L39.其他优化器，例如 Momentum 和 Adagrad 也使用槽.

See the source here if you're curious, it's actually quite readable: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adam.py#L39 . Other optimizers, such as Momentum and Adagrad use slots too.

必须先初始化这些变量，然后才能训练模型.

These variables must be initialized before you can train a model.

初始化变量的正常方法是调用 tf.initialize_all_variables()，它添加操作以初始化图中存在的变量调用时.

The normal way to initialize variables is to call tf.initialize_all_variables() which adds ops to initialize the variables present in the graph when it is called.

(另外:不像它的名字暗示的那样， initialize_all_variables() 不初始化任何东西，它只添加运行时初始化变量的操作.)

(Aside: unlike its name suggests, initialize_all_variables() does not initialize anything, it only add ops that will initialize the variables when run.)

你必须做的是调用 initialize_all_variables() 在你添加优化器之后:

What you must do is call initialize_all_variables() after you have added the optimizer:

...build your model...
# Add the optimizer
train_op = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
# Add the ops to initialize variables.  These will include 
# the optimizer slots added by AdamOptimizer().
init_op = tf.initialize_all_variables()

# launch the graph in a session
sess = tf.Session()
# Actually intialize the variables
sess.run(init_op)
# now train your model
for ...:
  sess.run(train_op)

这篇关于Tensorflow:使用 Adam 优化器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！