Tensorflow:恢复模型后如何更改优化器? [英] Tensorflow: How to change optimizer after restoring a model?

问题描述

我对TensorFlow还是很陌生，不知道如何解决有关更改优化器的简单问题.

I am quite new to TensorFlow and fail to know how to tackle this simple problem regarding changing optimizer.

例如，我想在恢复模型后将"AdamOptimizer"更改为"MomentumOptimizer".

For example, I'd like to change 'AdamOptimizer' to 'MomentumOptimizer' after restoring a model.

# optimizer = tf.train.AdamOptimizer(self.learning_rate)
optimizer = tf.train.MomentumOptimizer(learning_rate=self.learning_rate, momentum=0.8)

如果在优化程序后放置"self.saver = tf.train.Saver(tf.global_variables())"，则会引发NotFoundError.

If "self.saver = tf.train.Saver(tf.global_variables())" is put after the optimizer, a NotFoundError is raised like this.

NotFoundError(请参阅上面的回溯):在检查点中找不到键dynamic_seq2seq/decoder/attention/attention_layer/kernel/Momentum

如果在优化器之前放置"self.saver = tf.train.Saver(tf.global_variables())"，则在加载模型并开始对其进行训练后，将引发FailedPreconditionError.

And if "self.saver = tf.train.Saver(tf.global_variables())" is put before the optimizer, after model is loaded and starts to train, a FailedPreconditionError is raised like this.

FailedPreconditionError(请参阅上面的回溯):尝试使用未初始化的值dynamic_seq2seq/decoder/memory_layer/kernel/Momentum

在这些代码之后将调用"saver.restore(session，model_file)".

And "saver.restore(session, model_file)" is called after these codes.

在TensorFlow中，如何成功恢复模型而没有错误后如何更改优化器?

In TensorFlow, how to change optimizer after restoring a model successfully without the errors?

推荐答案

简而言之:

import tensorflow as tf

new_optimizer = tf.train.XXXOptimizer()

var_list = tf.trainable_variables()
# or if some variables are frozen
var_list = tf.tf.global_variables()

# Tell the new optimizer
new_optimizer._create_slots(var_list=var_list)

一些背景信息:

优化器可以定义所需的其他变量.他们为每个 可训练变量创建这些变量.这在两种情况下都是有问题的

Optimizers can define extra variables that they need. They create those variables for each trainable variable. This becomes problematic in two cases

1. 一个人在培训过程中更改了优化器，因为这是此问题的范围.
2. 一个人在训练期间解冻变量.言外之意，此变量之前被标记为trainable=False.这意味着先前使用的优化器(可以相同或不同)没有创建这些辅助变量.然后，它们不包含在检查点中.加载失败，因为假定它们存在.上面的示例可以变得更加简单:

1. One changes the optimizer during the training, as it is the scope of this question.
2. One unfreezes a variable during training. By implication, this variable was flagged as trainable=False before. This means that the previously used optimizer (can be the same or a different one) did not create these auxiliary variables. They are not included in the checkpoint then. Loading fails, because they are assumed to be present. The above example can become much simpler then:

trainable = False  # or True
layer = tf.layers.Layer(trainable=trainable)
outputs = layer(inputs)
new_optimizer._create_slots(var_list=layer.variables)

以上内容适用于所有优化器，因为按照惯例，如果要创建其他变量，则每个tf.train.XXXOptimizer都必须实现_create_slots().但是，这不是有史以来最好的解决方案，因为它使用了专用API.

The above works for all optimizers, because by convention, every tf.train.XXXOptimizer has to implement _create_slots() if additional variables shall be created. It is not the best solution ever, however, because it uses a private API.

这篇关于Tensorflow:恢复模型后如何更改优化器?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！