如何在Tensorflow中减少经过训练的模型中的内核/过滤器数量? [英] How to reduce the number of kernels/filters in a trained model in Tensorflow?

问题描述

如果我有一个训练有素的模型，那么我想在其中重新训练同一个模型，而从现有模型中删除了很少的过滤器/内核.例如

If I have a trained model, where I want to retrain the same model, with few filters/kernel removed from the existing model. e.g.

conv1 = tf.get_variable('conv1_1', shape=(11, 11, 3, 64), initializer=tf.contrib.layers.xavier_initializer()),

并且我想调整此张量的大小，使其具有(11，11，3，20)的形状，但名称和位置相同，表示变量完全相同.预先感谢您的帮助.

and I want to resize this tensor such that it has the shape of (11, 11, 3, 20) but the same name and position, mean exactly the same variable. Advance thanks for the help.

我尝试了tf.reshape，但是它给我错误，导致a和b中的元素数量不匹配 我也尝试过tf.assign(a,b, validate_shape=false)

I have tried tf.reshape but it gives me error of not matching the number of elements in a and b I have also tried tf.assign(a,b, validate_shape=false)

self.weights = {
    'conv1_': tf.get_variable('conv1_l1', shape=(11, 11, 3, 64), initializer=tf.contrib.layers.xavier_initializer()),
    'conv2_': tf.get_variable('conv2_l1', shape=(7, 7, 64, 128), initializer=tf.contrib.layers.xavier_initializer())
}

推荐答案

根本上，您正在更改模型的参数数量.当您出发时

Fundamentally, you are changing the number of parameters of your model. As you go from

conv1 = tf.get_variable('conv1_1', shape=(11, 11, 3, 64), initializer=tf.contrib.layers.xavier_initializer())

到

conv2 = tf.get_variable('conv2_l1', shape=(11, 11, 3, 20), initializer=tf.contrib.layers.xavier_initializer())

您可以学习的参数从{内核x 64}变为{内核x 20} 这将要求您重新训练网络并学习其新的权重.

Your learnable parameters goes from {Kernel x 64} to {Kernel x 20} This will require you to re-train the network and learn its new weights.

但是，这是一个普遍的问题，已经演变成一个研究领域.为此提出了许多方法，例如权重的低秩近似(Denton等人，2014； Lebedev等人，2014)，权重量化(Courbariaux等人，2016； Rastegari等人，2016)，知识蒸馏(Hinton等，2014; Romeroet等，2015)和网络修剪(Han等，2015; Li等，2017)，其中网络修剪由于其出色的竞争性能和兼容性而备受关注.

However, this is a common problem that has evolved into a research area. Many methods have been proposed for this such as low-rank approximation of weights (Denton et al., 2014; Lebedev et al., 2014), weight quantization(Courbariaux et al., 2016; Rastegari et al., 2016), knowledge distillation (Hinton et al., 2014; Romeroet al., 2015) and network pruning (Han et al., 2015; Li et al., 2017), among which network pruning has gained notable attention due to their competitive performance and compatibility.

要探索的参考文献:

1. Emily L Denton，Wojciech Zaremba，Joan Bruna，Yann LeCun和Rob Fergus.利用卷积网络内的线性结构进行有效评估. InNIPS，2014年.
2. Vadim Lebedev，Yaroslav Ganin，Maksim Rakhuba，Ivan Oseledets和Victor Lempitsky.使用微调cp分解加速卷积神经网络.ICLR，2014年.
3. Matthieu Courbariaux，Itay Hubara，Daniel Soudry，Ran El-Yaniv和Yoshua Bengio. Binarized神经网络:使用限制为+ 1or-1的权重和激活来训练深度神经网络.arXiv预印本arXiv:1602.02830，2016
4. Geoffrey Hinton，Oriol Vinyals和Jeff Dean.在神经网络中提取知识.NIPSWorkshop，2014年.

这篇关于如何在Tensorflow中减少经过训练的模型中的内核/过滤器数量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！