张量流中的 tf.GraphKeys.TRAINABLE_VARIABLES 和 tf.GraphKeys.UPDATE_OPS 有什么区别? [英] What's the differences between tf.GraphKeys.TRAINABLE_VARIABLES and tf.GraphKeys.UPDATE_OPS in tensorflow?

问题描述

这里是tensorflow中tf.GraphKeys的doc，比如TRAINABLE_VARIABLES:将被优化器训练的变量对象的子集.

Here is doc of tf.GraphKeys in tensorflow, such as TRAINABLE_VARIABLES: the subset of Variable objects that will be trained by an optimizer.

而且我知道 tf.get_collection()，它可以找到一些你想要的张量.

And i know tf.get_collection(), which can find some tensor that you want.

使用tensorflow.contrib.layers.batch_norm()时，参数updates_collections默认为GraphKeys.UPDATE_OPS.

我们如何理解这些集合，以及它们之间的区别.

How can we understand those collections, and difference in them.

此外，我们可以在 ops.py 中找到更多.

Besides, we can find more in ops.py.

推荐答案

这是两个不同的东西.

TRAINABLE_VARIABLES 是变量或训练参数的集合，在最小化损失时应该修改这些参数.例如，这些可以是确定网络中每个节点执行的功能的权重.

TRAINABLE_VARIABLES is the collection of variables or training parameters which should be modified when minimizing the loss. For example, these can be the weights determining the function performed by each node in the network.

如何将变量添加到此集合中?这会在您使用 tf.get_variable 定义新变量时自动发生，除非您指定

How do variables get added to this collection? This happens automatically when you define a new variable with tf.get_variable, unless you specify

tf.get_variable(..., trainable=False)

您希望什么时候变量不可训练?这种情况不时发生.例如，有时您会想要使用两步法，首先在大型通用数据集上训练整个网络，然后在与您的问题特别相关的较小数据集上微调网络.在这种情况下，您可能只想微调网络的一部分，例如最后一层.将某些变量指定为不可训练是实现此目的的方法之一.

When would you want a variable to be untrainable? This happens from time to time. For example, occasionally you will want to use a two-step approach in which you first train the entire network on a large, generic dataset, then fine-tune the network on a smaller dataset which is specifically related to your problem. In such cases, you might want to fine-tune only part of the network, e.g., the last layer. Specifying some variables as untrainable is one of the ways to do this.

UPDATE_OPS 是 ops(图运行时执行的操作，如乘法、ReLU 等)的集合，而不是变量.具体来说，这个集合维护了一个需要在每个训练步骤之前运行的操作列表.

UPDATE_OPS is a collection of ops (operations performed when the graph runs, like multiplication, ReLU, etc.), not variables. Specifically, this collection maintains a list of ops which need to run before each training step.

如何将操作添加到此集合中?根据定义，update_ops 发生在通过损失最小化的常规训练流程之外，因此通常您只会在特殊情况下向此集合添加操作.例如，在执行批量归一化时，您希望在每个训练步骤之前重新计算批量均值和方差，这就是它的完成方式.使用 tf.contrib.layers.batch_norm 进行批量标准化的机制在 这篇文章.

How do ops get added to this collection? By definition, update_ops occur outside the regular flow of training by loss minimization, so generally you will be adding ops to this collection only under special circumstances. For example, when performing batch normalization, you want to recompute the batch mean and variance before each training step, and this is how it's done. The mechanics of batch normalization using tf.contrib.layers.batch_norm are described in more detail in this article.

这篇关于张量流中的 tf.GraphKeys.TRAINABLE_VARIABLES 和 tf.GraphKeys.UPDATE_OPS 有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！