在 Tensorflow 中恢复作为新模型子集的变量? [英] Restore variables that are a subset of new model in Tensorflow?

问题描述

我正在做一个通过 Tensorflow 提升(4 层 DNN 到 5 层 DNN)的例子.我正在使用保存会话并在 TF 中恢复，因为 TF tute 中有一个简短的段落:'例如，你可能已经训练了一个 4 层的神经网络，而你现在想要训练一个 5 层的新模型，将之前训练模型的 4 层的参数恢复到新模型的前 4 层.'，其中 tensorflow tute 在 https://www.tensorflow.org/how_tos/variables/.

但是我发现没有人问过checkpoint保存4层参数的时候怎么用restore，但是我们需要把它放到5层，提红旗.

用真正的代码制作这个，我制作了

 with tf.name_scope('fcl1'):hidden_​​1 =fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)使用 tf.name_scope('fcl2'):hidden_​​2 =fully_connected_layer(hidden_​​1, num_hidden, num_hidden)使用 tf.name_scope('fclf'):hidden_​​final =fully_connected_layer(hidden_​​2, num_hidden, num_hidden)使用 tf.name_scope('outputl'):输出 = full_connected_layer(hidden_​​final，num_hidden，train_data.num_classes，tf.identity)输出 = tf.nn.softmax(输出)使用 tf.name_scope('boosting'):boosts = full_connected_layer(输出，train_data.num_classes，train_data.num_classes，tf.identity)

其中变量在(或从)'fcl1' - 所以我有'fcl1/Variable'和'fcl1/Variable_1'的权重和偏差 - 'fcl2'，'fclf'和'outputl'由saver.save() 在脚本中没有增强"层.但是，由于我们现在有 'boosting' 层，saver.restore(sess, "saved_models/model_list.ckpt") 无法正常工作

NotFoundError:在检查点中找不到密钥提升/Variable_1

我真的很希望听到这个问题.谢谢你.下面的代码是我遇到麻烦的代码的主要部分.

def full_connected_layer(inputs, input_dim, output_dim, 非线性=tf.nn.relu):权重 = tf.Variable(tf.truncated_normal([input_dim, output_dim], stddev=2./(input_dim + output_dim)**0.5),'权重')偏差 = tf.Variable(tf.zeros([output_dim]), '偏差')输出 = 非线性(tf.matmul(输入，权重)+ 偏差)返回输出输入 = tf.placeholder(tf.float32, [None, train_data.inputs.shape[1]], 'inputs')目标 = tf.placeholder(tf.float32, [None, train_data.num_classes], 'targets')使用 tf.name_scope('fcl1'):hidden_​​1 =fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)使用 tf.name_scope('fcl2'):hidden_​​2 =fully_connected_layer(hidden_​​1, num_hidden, num_hidden)使用 tf.name_scope('fclf'):hidden_​​final =fully_connected_layer(hidden_​​2, num_hidden, num_hidden)使用 tf.name_scope('outputl'):输出 = full_connected_layer(hidden_​​final，num_hidden，train_data.num_classes，tf.identity)使用 tf.name_scope('error'):错误 = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(输出，目标))使用 tf.name_scope('accuracy'):精度 = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(outputs, 1), tf.argmax(targets, 1)),tf.float32))使用 tf.name_scope('train'):train_step = tf.train.AdamOptimizer().minimize(error)init = tf.global_variables_initializer()保护程序 = tf.train.Saver()使用 tf.Session() 作为 sess:sess.run(初始化)saver.restore(sess, "saved_models/model.ckpt")打印(模型恢复")print("优化开始！")对于范围内的 e(training_epochs):...#Save 模型 - 保存会话save_path = saver.save(sess, "saved_models/model.ckpt")### 我曾经使用 var_list 保存了变量，但效果不佳...print("模型保存在文件中:%s" % save_path)

为了清楚起见，检查点文件有

fcl1/变量:0fcl1/Variable_1:0fcl2/变量:0fcl2/Variable_1:0fclf/变量:0fclf/Variable_1:0输出/变量:0输出l/Variable_1:0

由于原来的 4 层模型没有 'boosting' 层.

解决方案

在这种情况下，从检查点读取用于提升的值看起来不正确，我认为这不是您想要做的.显然，您遇到了错误，因为在恢复变量时，您首先要捕获模型中所有变量的列表，然后在检查点中查找相应的变量，而检查点没有这些变量.

您可以通过定义模型变量的子集来仅恢复模型的一部分.例如，您可以使用 tf.slim 库来实现.获取模型中的变量列表:

variables = slim.get_variables_to_restore()

现在变量是张量列表，但对于每个元素，您都可以访问其名称属性.使用它，您可以指定您只想恢复除提升以外的层，例如:

variables_to_restore = [v for v in variables if v.name.split('/')[0]!='boosting']model_path = '你的/模型/路径'saver = tf.train.Saver(variables_to_restore)使用 tf.Session() 作为 sess:saver.restore(sess, model_path)

通过这种方式，您将恢复 4 个图层.从理论上讲，您可以尝试通过创建另一个服务器来从检查点捕获一个变量的值，该服务器只会增加变量列表并从检查点重命名所选变量，但我真的不认为这是您在这里需要的.

由于这是您模型的自定义层，并且您在任何地方都没有此变量，因此只需在您的工作流程中对其进行初始化，而不是尝试导入它.例如，您可以通过在调用 full_connected 函数时传递此参数来实现:

weights_initializer = slim.variance_scaling_initializer()

不过，您需要自己检查详细信息，因为我不确定您的导入是什么以及您在这里使用的是哪个函数.

通常，我建议您查看 slim 库，这将使您更轻松地定义层的模型和作用域(而不是通过 with 定义它，您可以在调用函数时传递作用域参数).苗条的看起来像这样:

boost = slim.fully_connected(input, number_of_outputs, activation_fn=None, scope='boosting', weights_initializer=slim.variance_scaling_initializer())

I am doing an example for boosting(4 layers DNN to 5 layers DNN) via Tensorflow. I am making it with save session and restore in TF because there is a brief paragraph in TF tute: 'For example, you may have trained a neural net with 4 layers, and you now want to train a new model with 5 layers, restoring the parameters from the 4 layers of the previously trained model into the first 4 layers of the new model.', where tensorflow tute inspires in https://www.tensorflow.org/how_tos/variables/.

However, I found that nobody has asked how to use 'restore' when the checkpoint saves parameters of 4 layers but we need to put that into 5 layers, raising a red flag.

Making this in real code, I made

with tf.name_scope('fcl1'):
    hidden_1 = fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)            
with tf.name_scope('fcl2'):
    hidden_2 = fully_connected_layer(hidden_1, num_hidden, num_hidden)                
with tf.name_scope('fclf'):
    hidden_final = fully_connected_layer(hidden_2, num_hidden, num_hidden)    
with tf.name_scope('outputl'):
    outputs = fully_connected_layer(hidden_final, num_hidden, train_data.num_classes, tf.identity)
    outputs = tf.nn.softmax(outputs)
with tf.name_scope('boosting'):
    boosts = fully_connected_layer(outputs, train_data.num_classes, train_data.num_classes, tf.identity)

where variables inside(or called from) 'fcl1' - so that I have 'fcl1/Variable' and 'fcl1/Variable_1' for weight and bias - 'fcl2', 'fclf', and 'outputl' are stored by saver.save() in the script without 'boosting' layer. However, as we now have 'boosting' layer, saver.restore(sess, "saved_models/model_list.ckpt") does not work as

NotFoundError: Key boosting/Variable_1 not found in checkpoint

I really hope to hear about this problem. Thank you. Below code is the main part of the code I am in trouble.

def fully_connected_layer(inputs, input_dim, output_dim, nonlinearity=tf.nn.relu):
    weights = tf.Variable(
        tf.truncated_normal(
            [input_dim, output_dim], stddev=2. / (input_dim + output_dim)**0.5), 
        'weights')
    biases = tf.Variable(tf.zeros([output_dim]), 'biases')
    outputs = nonlinearity(tf.matmul(inputs, weights) + biases)    

    return outputs

inputs = tf.placeholder(tf.float32, [None, train_data.inputs.shape[1]], 'inputs')
targets = tf.placeholder(tf.float32, [None, train_data.num_classes], 'targets')

with tf.name_scope('fcl1'):
    hidden_1 = fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)            
with tf.name_scope('fcl2'):
    hidden_2 = fully_connected_layer(hidden_1, num_hidden, num_hidden)                
with tf.name_scope('fclf'):
    hidden_final = fully_connected_layer(hidden_2, num_hidden, num_hidden)    
with tf.name_scope('outputl'):
    outputs = fully_connected_layer(hidden_final, num_hidden, train_data.num_classes, tf.identity)

with tf.name_scope('error'):    
    error = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(outputs, targets))
with tf.name_scope('accuracy'):
    accuracy = tf.reduce_mean(tf.cast(
        tf.equal(tf.argmax(outputs, 1), tf.argmax(targets, 1)), 
        tf.float32))
with tf.name_scope('train'):
    train_step = tf.train.AdamOptimizer().minimize(error)

init = tf.global_variables_initializer()  
saver = tf.train.Saver()

with tf.Session() as sess:
    sess.run(init)
    saver.restore(sess, "saved_models/model.ckpt")
    print("Model restored")

    print("Optimization Starts!")
    for e in range(training_epochs):
        ...

   #Save model - save session        
    save_path = saver.save(sess, "saved_models/model.ckpt")
    ### I once saved the variables using var_list, but didn't work as well...
    print("Model saved in file: %s" % save_path)

For clarity, the checkpoint file has

fcl1/Variable:0

fcl1/Variable_1:0

fcl2/Variable:0

fcl2/Variable_1:0

fclf/Variable:0

fclf/Variable_1:0

outputl/Variable:0

outputl/Variable_1:0

As the original 4 layers model does not have 'boosting' layer.

解决方案

It doesn't look right to read values for boosting from the checkpoint in this case and I think that's not what you want to do. Obviously you're getting error, since while restoring the variables you are first catching the list of all of the variables in your model and then you look for corresponding variables in your checkpoint, which doesn't have them.

You can restore only part of your model by defining a subset of your model variables. For example you can do it using tf.slim library. Getting the list of variables in your models:

variables = slim.get_variables_to_restore()

Now variables is a list of tensors, but for each element you can access its name attribute. Using that you can specify that you only want to restore layers other than boosting, e.g.:

variables_to_restore = [v for v in variables if v.name.split('/')[0]!='boosting'] 
model_path = 'your/model/path'

saver = tf.train.Saver(variables_to_restore)

with tf.Session() as sess:
    saver.restore(sess, model_path)

This way you will have your 4 layers restored. Theoretically you could try to catch values of one of your variables from checkpoint by creating another server that will only have boosting in variables list and renaming the chosen variable from the checkpoint, but I really don't think it's what you need here.

Since this is a custom layer for your model and you don't have this variable anywhere, just initialize it within your workflow instead of trying to import it. You can do for example by passing this argument while calling a function fully_connected:

weights_initializer = slim.variance_scaling_initializer()

You need to check details yourself though, since I'm not sure what your imports are and which function are you using here.

Generally I'd advice you to take a look at slim library, which will make it easier for you to define a model and scopes for layers (instead of defining it by with you can rather pass a scope argument while calling a function). It would look something like that with slim:

boost = slim.fully_connected(input, number_of_outputs, activation_fn=None, scope='boosting', weights_initializer=slim.variance_scaling_initializer())

这篇关于在 Tensorflow 中恢复作为新模型子集的变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！