如何获得keras中的梯度? [英] How to obtain the gradients in keras?

问题描述

我正在尝试调试我构建的 keras 模型.似乎我的梯度正在爆炸，或者被 0 或类似的除法.能够在它们通过网络反向传播时检查各种梯度会很方便.类似以下的内容将是理想的:

I am attempting to debug a keras model that I have built. It seems that my gradients are exploding, or there is a division by 0 or some such. It would be convenient to be able to inspect the various gradients as they back-propagate through the network. Something like the following would be ideal:

model.evaluate(np.array([[1,2]]), np.array([[1]])) #gives the loss
model.evaluate_gradient(np.array([[1,2]]), np.array([[1]]), layer=2) #gives the doutput/dloss at layer 2 for the given input
model.evaluate_weight_gradient(np.array([[1,2]]), np.array([[1]]), layer=2) #gives the dweight/dloss at layer 2 for the given input

推荐答案

您需要创建一个符号化的 Keras 函数，将输入/输出作为输入并返回梯度.这是一个工作示例:

You need to create a symbolic Keras function, taking the input/output as inputs and returning the gradients. Here is a working example :

import numpy as np
import keras
from keras import backend as K

model = keras.Sequential()
model.add(keras.layers.Dense(20, input_shape = (10, )))
model.add(keras.layers.Dense(5))
model.compile('adam', 'mse')

dummy_in = np.ones((4, 10))
dummy_out = np.ones((4, 5))
dummy_loss = model.train_on_batch(dummy_in, dummy_out)

def get_weight_grad(model, inputs, outputs):
    """ Gets gradient of model for given inputs and outputs for all weights"""
    grads = model.optimizer.get_gradients(model.total_loss, model.trainable_weights)
    symb_inputs = (model._feed_inputs + model._feed_targets + model._feed_sample_weights)
    f = K.function(symb_inputs, grads)
    x, y, sample_weight = model._standardize_user_data(inputs, outputs)
    output_grad = f(x + y + sample_weight)
    return output_grad


def get_layer_output_grad(model, inputs, outputs, layer=-1):
    """ Gets gradient a layer output for given inputs and outputs"""
    grads = model.optimizer.get_gradients(model.total_loss, model.layers[layer].output)
    symb_inputs = (model._feed_inputs + model._feed_targets + model._feed_sample_weights)
    f = K.function(symb_inputs, grads)
    x, y, sample_weight = model._standardize_user_data(inputs, outputs)
    output_grad = f(x + y + sample_weight)
    return output_grad


weight_grads = get_weight_grad(model, dummy_in, dummy_out)
output_grad = get_layer_output_grad(model, dummy_in, dummy_out)

我编写的第一个函数返回模型中的所有梯度，但扩展它并不困难，因此它支持层索引.但是，这可能很危险，因为模型中没有权重的任何层都将被此索引忽略，并且您最终会在模型和梯度中使用不同的层索引.
我编写的第二个函数返回给定层输出的梯度，在那里，索引与模型中的相同，因此使用它是安全的.

The first function I wrote returns all the gradients in the model but it wouldn't be difficult to extend it so it supports layer indexing. However, it's probably dangerous because any layer without weights in the model will be ignored by this indexing and you would end up with different layer indexing in the model and the gradients.
The second function I wrote returns the gradient at a given layer's output and there, the indexing is the same as in the model, so it's safe to use it.

注意:这适用于 Keras 2.2.0，而不适用于 Keras 2.2.0，因为此版本包括对 keras.engine

Note : This works with Keras 2.2.0, not under, as this release included a major refactoring of keras.engine

这篇关于如何获得keras中的梯度?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！