获取给定优化器在 Tensorflow 中最小化的损失 [英] Get the loss that a given optimizer is minimizing in Tensorflow

问题描述

我正在为我的 tensorflow 工作区在单元测试系统中工作，我想知道是否有任何方法或属性，给定一个带有优化器操作的图(在调用 .minimize() 之后)，以获得最终损失它正在优化的张量及其控制的变量.

I am working in a unit test system for my tensorflow workspace and I would like to know if there is any method or attribute, given a graph with an optimizer operation (after calling .minimize()), to obtain the final loss tensor that it is optimizing and the variables that it controls.

例如，如果我调用 train_op = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy) 我想检索 cross_entropy 只能访问 train_op.

For example if I call train_op = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy) I would like to retrieve cross_entropy only having access to train_op.

我可以访问 train_op 对象，我只想知道它引用了哪个损失以及哪些变量控制.

I have access to the train_op object, I only want to know to which loss it is referenced and which variables controls.

推荐答案

很简单:

def build_graph():
  cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(...)
  train_op = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
  return cross_entropy, train_op    # both tensorflow OPs

cross_entropy, train_op = build_graph()

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  # Evaluate and return cross_entropy, and execute the train_op (update the weights)
  result = sess.run([cross_entropy, train_op], feed_dict={...})

  print(result[0])   # The value of the cross entropy loss function

这里有很多优秀的教程:https://github.com/aymericdamien/TensorFlow-Examples

There are many excellent tutorials here: https://github.com/aymericdamien/TensorFlow-Examples

您会发现他们在完整的工作模型中完全做到了这一点.

You'll find them doing exactly this in full working models.

如果您无权访问张量，那么您可以在图中按名称查找它:

If you don't have access to the tensor, then you can look it up in the graph by name as such:

tf.get_default_graph().get_tensor_by_name("example:0")

参见这个问题:Tensorflow:如何通过名称获取张量?

请注意，如果您没有很好地命名张量，这将是一个巨大的痛苦，因此，这是为您的张量命名的众多充分理由之一.张量的默认名称将使用对操作的一些引用、冒号、索引号，例如第三次添加操作的add:2".

Note that if you didn't name your tensors well, this is going to be a royal pain in the rear, so, it's one of many good reasons to name your tensors well. The default name of a tensor will use some reference to the operation, colon, an index number, such as "add:2" for the 3rd add operation.

您可以使用以下命令获取图中所有张量的列表:

You can get a list of all tensors in a graph with:

[n.name for n in tf.get_default_graph().as_graph_def().node]

该代码是从这个问题复制而来的:在 Tensorflow 中，获取图中所有张量的名称

That code is copied from from this question: In Tensorflow, get the names of all the Tensors in a graph

在评论中回答这个跟进问题:

Responding to this follow up question in comments:

我想知道哪个是 train_op 优化而不是必须用特定的名称命名它们.所以给定一个 train_op 对象，有没有办法检索张量(或张量的名称)这代表 train_op 最小化的最后一个值?我需要它因为我正在自动化一组单元测试，所以如果我插入一个tensorflow 模型到我的系统它会自动找到，给定优化器，代表损失的张量(这样我可以自动执行梯度检查).

I would like to know which one is the train_op optimizing without having to name them with a specific name. So given a train_op object, is there any way of retrieving the tensor (or the name of the tensor) which represents the last value that train_op is minimizing? I need it because I am automatizing a set of unit tests so that if I plug a tensorflow model to my system it automatically finds, given the optimizers, the tensors that represent the loss (that way I can automatically perform gradient checks).

作为我研究的一部分，我编写了一个梯度下降优化器.以下是您可能会考虑的一些想法:

I have coded a gradient descent optimizer as part of my research. Here are a few ideas you might consider:

1) 这是我执行相同操作时遵循的优化器的链接:https://github.com/openai/iaf/blob/master/tf_utils/adamax.py 那就是在 python 中实现 AdaMax.您会对 _apply_dense() 感兴趣，它采用梯度及其变量并执行更新.每个可训练变量都会调用它.请注意，tensorflow 中的大多数优化器都是用 C 编写的，而不是使用 python 接口.所以我不确定这是否会有所帮助，但更好地理解这个过程并不是一件坏事.

1) Here's a link to the optimizer I followed when I did the same: https://github.com/openai/iaf/blob/master/tf_utils/adamax.py That's implementing AdaMax in python. You'll be interested in _apply_dense() which takes a gradient and its variable and performs the update. It's called for each trainable variable. Note that most optimizers in tensorflow are coded in C, not using the python interface. So I'm not sure if this will help or not, but understanding the process better can't be a bad thing.

2) 你可以得到任何变量相对于任何其他变量的梯度.因此，您可以使用 tf.trainable_variables() 获取可训练变量的集合，然后调用 tf.gradients 来获取可训练变量相对于损失函数的梯度.不过，您需要为此使用损失函数，而不是火车 OP.我希望您可以从优化器中自动找到损失.

2) You can get the gradient of any variable with respect to any other variable. So you could grab the collection of trainable variables with tf.trainable_variables() and then call tf.gradients to get the gradients of the trainable variables with respect to the loss function. You would need the loss function for this rather than the train OP though. I expect you can find the loss automatically from the optimizer.

如果您只是想从火车 OP 中找到损失函数，您可能会按照本问题中描述的图形依赖关系找到您需要的:如何列出一个节点依赖的所有 Tensorflow 变量?

If you're just trying to find the loss function from the train OP you might find what you need by following the graph dependencies as is described in this question: How can I list all Tensorflow variables a node depends on?

这是我以前使用的一种方法来获取每个变量及其输入和输出的列表.我怀疑你能弄清楚如何遍历这个数据结构来找到你需要的东西.

Here's a way I've used before to get a list of each variable and its inputs and outputs. I suspect you could figure out how to traverse this datastructure to find what you need.

tf.get_default_graph().as_graph_def()

Out[6]: 
node {
  name: "x"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_FLOAT
        tensor_shape {
        }
        float_val: 42.0
      }
    }
  }
}

这篇关于获取给定优化器在 Tensorflow 中最小化的损失的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！