如何在Tensorflow中创建优化器 [英] How to create an optimizer in Tensorflow

问题描述

我想为Tensorflow上的网络编写新的优化算法.我希望实现 Levenberg Marquardt优化算法，该算法现在已从TF API中排除.我发现有关如何编写自定义优化器的文档不多，所以我问是否有人可以给我任何建议.谢谢.

I want to write a new optimization algorithm for my network on Tensorflow. I hope to implement the Levenberg Marquardt optimization algorithm, which now is excluded from TF API. I found poor documentation on how to write a custom optimizer, so i ask if someone can give my any advice. Thanks.

推荐答案

最简单的优化器示例可能是优化器类的实例.优化器基类文档说明了这些方法的作用.

The simplest example of an optimizer is probably the gradient descent optimizer. It shows how one creates an instance of the basic optimizer class. The optimizer base class documentation explains what the methods do.

优化器的python端将新节点添加到图中，以计算和应用反向传播的渐变.它提供传递给操作的参数，并对优化器进行一些高级管理.然后，您需要实际的应用"操作.

The python side of the optimizers adds new nodes to the graph that compute and apply the gradients being back-propagated. It supplies the parameters that get passed to the ops and does some of the high-level management of the optimizer. Then, you need the actual "Apply" op.

Ops同时具有python和C ++组件.编写培训操作与向TensorFlow添加操作的一般过程相同(但专门). /a>.

Ops have both a python and a C++ component. Writing a training op is the same (but specialized) as the general process of adding an Op to TensorFlow.

有关计算和应用渐变的一组训练操作示例，请参阅 python/training/training_ops.py -这是Python胶水实际的培训操作.请注意，这里的代码主要是关于形状推断的-计算将在C ++中进行.

For an example set of training ops that compute and apply gradients, see python/training/training_ops.py - this is the Python glue for the actual training ops. Note that the code here is mostly about shape inference - the computation is going to be in the C++.

应用渐变的实际数学运算由Op处理(回想一下，OP通常是用C ++编写的).在这种情况下，应用渐变操作在 core/kernels/training_ops.cc中定义.例如，您可以在其中看到ApplyGradientDescentOp的实现，该实现引用了函子ApplyGradientDescent:

The actual math for applying the gradients is handled by an Op (recalling that, in general, ops are written in C++). In this case, the apply gradients ops are defined in core/kernels/training_ops.cc. You can see, for example, the implementation of ApplyGradientDescentOp in there, which references a functor ApplyGradientDescent:

var.device(d) -= grad * lr();

Op本身的实现遵循增添-op文档中所述的任何其他op的实现.

The implementation of the Op itself follows the implementation of any other op as described in the adding-an-op docs.

这篇关于如何在Tensorflow中创建优化器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！