神经网络的显着性图(使用Keras) [英] Saliency maps of neural networks (using Keras)

问题描述

我有一个在Keras训练的完全连接的多层感知器.我为它提供了N维特征向量，并且它为输入向量预测了M个类中的一个.训练和预测效果很好.现在，我要分析输入特征向量的哪一部分实际上负责特定的类.
例如，假设有两个类A和B，以及一个输入向量f.向量f属于类A，并且网络可以正确预测-网络的输出为A=1 B=0.因为我了解一些领域知识，所以我知道整个f实际上对属于A的f不负责，而在f中仅某个部分负责.我想知道神经网络是否已经捕获了这一点.绘制与图像的对应关系，如果图像I中具有cat(具有一些草皮背景)，并且受过训练的网络可以正确预测该图像，则网络必须知道整个图像实际上不是cat；否则，网络必须知道整个图像实际上不是cat.网络内部知道cat在图像中的位置.类似地，在我的情况下，网络知道f的哪个部分使其属于类A.我想知道那是哪一部分.

I have a fully connected multilayer perceptron trained in Keras. I feed it an N-dimensional feature vector and it predicts one out of M classes for the input vector. The training and prediction is working well. Now I want to analyze what part of the input feature vector is actually responsible for a particular class.
For example, lets say there are two classes A and B , and an input vector f. The vector f belongs to class A and the network predicts it correctly - the output of the network is A=1 B=0. Because I have some domain knowledge, I know that the entire f is actually not responsible for f belonging to A, only a certain part inside f is responsible for that. I want to know if the neural network has captured that. Drawing a correspondence to images, if an image I has a cat in it (with some grassy background) and a trained network predicts that correctly, then the network must know that the entire image is actually not a cat; the network internally knows the location of the cat in the image. Similarly, in my case, the network knows what part of f makes it belong to class A. I want to know what part that is.

我到处搜索，并相信我想要做的就是为给定的输入为我的网络查找显着图.正确吗?
如果我对它的理解正确，那么显着性映射就是(change in output)/(change in input)，可以通过1次反向传播操作找到，因为我在该操作中找到了相对于输入的输出导数.
我在Keras中找到了用于执行此操作的以下代码段，但我不确定它是否正确:

I searched around, and believe what I want to do is called finding Saliency Maps for my network, for a given input. Is that correct?
If I've understood it correctly, Saliency Maps are simply (change in output)/(change in input), and can be found by simply 1 backpropagation operation where I find the derivative of output with respect to the input.
I found the following code snippet for doing this in Keras, but I'm not really sure if it is correct:

   inp = model.layers[0].get_input()
   outp = model.layers[-1].get_output()
   max_outp = T.max(outp, axis=1)
   saliency = theano.grad(max_outp.sum(), wrt=inp)

在上面的代码中，当计算梯度时，反向传播实际上是否在发生?输出是输入的非线性函数，因此查找渐变的唯一方法是进行反向传播.但是在上面的代码中，没有任何东西可以连接theano和网络，theano在这里如何感知"网络?据我所知，在使用Theano计算梯度时，我们首先根据输入和输出定义函数.因此，theano必须知道非线性函数是什么.我不认为上面的代码段是这样.

In the above code, when computing the gradient, is the backpropagation actually happening? The output is a non-linear function of the input, so the only way to find the gradient is to do backprop. But in the above code, there is nothing to connect theano and the network, how is theano "aware" of the network here? As far as I know, when computing gradients with Theano, we first define the function in terms of input and output. So theano has to know what that non-linear function is. I don't think that is true in the above snippet..

更新:上面的代码不起作用，因为我具有完全连接的MLP.它给出一个错误信息:密集对象没有get_output()".我有以下Keras函数，该函数计算给定输入的网络输出.我现在想用输入找到该函数的梯度:

Update: The above code doesn't work because I have a fully connected MLP. It gives an error saying "Dense object doesn't have get_output()" . I have the following Keras function, which computes output of network given input. I want to now find gradient of this function wrt the input:

    get_output = K.function([self.model.layers[0].input],[self.model.layers[-1].output])

推荐答案

我找到了解决方法:

    get_output = theano.function([model.layers[0].input],model.layers[-1].output,allow_input_downcast=True)
    fx = theano.function( [model.layers[0].input] ,T.jacobian(model.layers[-1].output.flatten(),model.layers[0].input), allow_input_downcast=True)
    grad = fx([input_feature])

这篇关于神经网络的显着性图(使用Keras)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！