如何获得喀拉斯山脉的梯度? [英] How to obtain the gradients in keras?

问题描述

我正在尝试调试已建立的keras模型.似乎我的梯度正在爆炸，或者被0除以此类推.当它们通过网络向后传播时，能够检查各种梯度将是很方便的.像下面这样的东西将是理想的:

I am attempting to debug a keras model that I have built. It seems that my gradients are exploding, or there is a division by 0 or some such. It would be convenient to be able to inspect the various gradients as they back-propagate through the network. Something like the following would be ideal:

model.evaluate(np.array([[1,2]]), np.array([[1]])) #gives the loss
model.evaluate_gradient(np.array([[1,2]]), np.array([[1]]), layer=2) #gives the doutput/dloss at layer 2 for the given input
model.evaluate_weight_gradient(np.array([[1,2]]), np.array([[1]]), layer=2) #gives the dweight/dloss at layer 2 for the given input

推荐答案

您需要创建一个符号性Keras函数，将输入/输出作为输入并返回渐变. 这是一个有效的示例:

You need to create a symbolic Keras function, taking the input/output as inputs and returning the gradients. Here is a working example :

import numpy as np
import keras
from keras import backend as K

model = keras.Sequential()
model.add(keras.layers.Dense(20, input_shape = (10, )))
model.add(keras.layers.Dense(5))
model.compile('adam', 'mse')

dummy_in = np.ones((4, 10))
dummy_out = np.ones((4, 5))
dummy_loss = model.train_on_batch(dummy_in, dummy_out)

def get_weight_grad(model, inputs, outputs):
    """ Gets gradient of model for given inputs and outputs for all weights"""
    grads = model.optimizer.get_gradients(model.total_loss, model.trainable_weights)
    symb_inputs = (model._feed_inputs + model._feed_targets + model._feed_sample_weights)
    f = K.function(symb_inputs, grads)
    x, y, sample_weight = model._standardize_user_data(inputs, outputs)
    output_grad = f(x + y + sample_weight)
    return output_grad


def get_layer_output_grad(model, inputs, outputs, layer=-1):
    """ Gets gradient a layer output for given inputs and outputs"""
    grads = model.optimizer.get_gradients(model.total_loss, model.layers[layer].output)
    symb_inputs = (model._feed_inputs + model._feed_targets + model._feed_sample_weights)
    f = K.function(symb_inputs, grads)
    x, y, sample_weight = model._standardize_user_data(inputs, outputs)
    output_grad = f(x + y + sample_weight)
    return output_grad


weight_grads = get_weight_grad(model, dummy_in, dummy_out)
output_grad = get_layer_output_grad(model, dummy_in, dummy_out)

我编写的第一个函数返回了模型中的所有渐变，但是扩展它并不难，因此它支持图层索引.但是，这可能很危险，因为此索引将忽略模型中没有权重的任何图层，并且最终在模型和渐变中将具有不同的图层索引.
我编写的第二个函数在给定层的输出处返回渐变，在那里，索引与模型中的相同，因此可以安全地使用它.

The first function I wrote returns all the gradients in the model but it wouldn't be difficult to extend it so it supports layer indexing. However, it's probably dangerous because any layer without weights in the model will be ignored by this indexing and you would end up with different layer indexing in the model and the gradients.
The second function I wrote returns the gradient at a given layer's output and there, the indexing is the same as in the model, so it's safe to use it.

注意:它与Keras 2.2.0兼容，但不兼容，因为此发行版包括对keras.engine

Note : This works with Keras 2.2.0, not under, as this release included a major refactoring of keras.engine

这篇关于如何获得喀拉斯山脉的梯度?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！