如何在TensorFlow中应用梯度裁剪? [英] How to apply gradient clipping in TensorFlow?

问题描述

考虑示例代码

我想知道如何在RNN上的这个网络上应用梯度剪切，而梯度可能会爆炸.

I would like to know How to apply gradient clipping on this network on the RNN where there is a possibility of exploding gradients.

tf.clip_by_value(t, clip_value_min, clip_value_max, name=None)

这是一个可以使用的示例，但是我在哪里介绍呢? 按照RNN的定义

This is an example that could be used but where do I introduce this ? In the def of RNN

    lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
    # Split data because rnn cell needs a list of inputs for the RNN inner loop
    _X = tf.split(0, n_steps, _X) # n_steps
tf.clip_by_value(_X, -1, 1, name=None)

但这是没有意义的，因为张量_X是输入，而不是grad，要裁剪的内容是什么?

But this doesn't make sense as the tensor _X is the input and not the grad what is to be clipped?

我是否需要为此定义自己的优化器，还是有一个更简单的选项?

Do I have to define my own Optimizer for this or is there a simpler option?

推荐答案

在计算梯度之后，但在应用梯度更新模型参数之前，需要进行梯度修剪.在您的示例中，这两种情况均由AdamOptimizer.minimize()方法处理.

Gradient clipping needs to happen after computing the gradients, but before applying them to update the model's parameters. In your example, both of those things are handled by the AdamOptimizer.minimize() method.

要剪切渐变，您需要按照

In order to clip your gradients you'll need to explicitly compute, clip, and apply them as described in this section in TensorFlow's API documentation. Specifically you'll need to substitute the call to the minimize() method with something like the following:

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
gvs = optimizer.compute_gradients(cost)
capped_gvs = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gvs]
train_op = optimizer.apply_gradients(capped_gvs)

这篇关于如何在TensorFlow中应用梯度裁剪?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！