Tensorflow负载预训练模型使用不同的优化器 [英] Tensorflow load pre-trained model use different optimizer

问题描述

我想加载一个预先训练的模型(由AdadeltaOptimizer优化)，并继续使用SGD(GradientDescentOptimizer)进行训练.使用 tensorlayer API

I want to load a pre-trained model (optimized by AdadeltaOptimizer) and continue training with SGD (GradientDescentOptimizer). The models are saved and loaded with tensorlayer API:

保存模型:

import tensorlayer as tl
tl.files.save_npz(network.all_params,
                  name=model_dir + "model-%d.npz" % global_step)

加载模型:

load_params = tl.files.load_npz(path=resume_dir + '/', name=model_name)
tl.files.assign_params(sess, load_params, network)

如果我继续使用adadelta进行训练，则训练损失(交叉熵)看起来很正常(从加载模型的接近值开始).但是，如果我将优化器更改为SGD，则训练损失将与新初始化的模型一样大.

If I continue training with adadelta, the training loss (cross entropy) looks normal (start at a close value as the loaded model). However, if I change the optimizer to SGD, the training loss would be as large as a newly initialized model.

我看了tl.files.save_npz中的model-xxx.npz文件.它仅将所有模型参数另存为ndarray.我不确定这里如何涉及优化器或学习率.

I took a look at the model-xxx.npz file from tl.files.save_npz. It only saves all model parameters as ndarray. I'm not sure how the optimizer or learning rate is involved here.

推荐答案

您可能必须将张量导入到变量中，该变量是先前输入到您的Adam Optimizer中的损失函数/交叉熵 >.现在，只需将其通过您的SGD优化器输入即可.

You probably would have to import the tensor into a variable which is the loss function/cross-entropy that feeds into your Adam Optimizer previously. Now, just feed it through your SGD optimizer instead.

saver = tf.train.import_meta_graph('filename.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))
graph = tf.get_default_graph()
cross_entropy = graph.get_tensor_by_name("entropy:0") #Tensor to import

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cross_entropy)

在这种情况下，在训练我的训练前模型之前，我已经标记了交叉熵张量，就像这样

In this case, I have tagged the cross-entropy Tensor before training my pre-train model with the name entropy, as such

tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv), name = 'entropy')

如果您无法更改预训练模型，则可以从graph获取模型中的张量列表(在导入后)，然后推断出所需的张量.我没有使用Tensorlayer的经验，因此本指南旨在提供更多的理解.您可以看一下 Tensorlayer层，他们应该解释如何获得张量由于Tensorlayer是建立在Tensorflow之上的，因此大多数功能仍然应该可用.

If you are unable to make changes to your pretrain model, you can obtain the list of Tensors in your model(after you have imported it) from graph and deduce which Tensor you require. I have no experience with Tensorlayer, so this guide is to provide more of an understanding. You can take a look at Tensorlayer-Layers, they should explain how to obtain your Tensor. As Tensorlayer is built on top of Tensorflow, most of the functions should still be available.

这篇关于Tensorflow负载预训练模型使用不同的优化器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！