如何在Tensorflow中仅使用Python制作自定义激活功能? [英] How to make a custom activation function with only Python in Tensorflow?

问题描述

假设您需要创建一个激活功能，而仅使用预定义的张量流构建块是不可能的，那该怎么办?

Suppose you need to make an activation function which is not possible using only pre-defined tensorflow building-blocks, what can you do?

因此在Tensorflow中可以创建自己的激活功能.但这很复杂，您必须使用C ++编写它并重新编译整个tensorflow

So in Tensorflow it is possible to make your own activation function. But it is quite complicated, you have to write it in C++ and recompile the whole of tensorflow [1] [2].

有没有更简单的方法?

推荐答案

是的！

信用: 很难找到信息并使之正常工作，但这是从找到的原理和代码中复制的示例 和此处.

Credit: It was hard to find the information and get it working but here is an example copying from the principles and code found here and here.

要求: 在我们开始之前，要使此方法成功必须满足两个条件.首先，您需要能够将激活作为函数写入numpy数组.其次，您必须能够将该函数的派生形式编写为Tensorflow中的函数(更简单)，或者在最坏的情况下将其编写为numpy数组中的函数.

Requirements: Before we start, there are two requirement for this to be able to succeed. First you need to be able to write your activation as a function on numpy arrays. Second you have to be able to write the derivative of that function either as a function in Tensorflow (easier) or in the worst case scenario as a function on numpy arrays.

文字激活功能:

所以让我们举一个我们想使用激活函数的函数为例:

So let's take for example this function which we would want to use an activation function:

def spiky(x):
    r = x % 1
    if r <= 0.5:
        return r
    else:
        return 0

其外观如下:

第一步是使其成为numpy函数，这很容易:

The first step is making it into a numpy function, this is easy:

import numpy as np
np_spiky = np.vectorize(spiky)

现在我们应该写它的派生词.

Now we should write its derivative.

激活梯度: 在我们的情况下很容易，如果x mod 1 ＜1，则为1. 0.5和0，否则.所以:

Gradient of Activation: In our case it is easy, it is 1 if x mod 1 < 0.5 and 0 otherwise. So:

def d_spiky(x):
    r = x % 1
    if r <= 0.5:
        return 1
    else:
        return 0
np_d_spiky = np.vectorize(d_spiky)

现在要使用TensorFlow函数来解决这个难题.

Now for the hard part of making a TensorFlow function out of it.

对tensorflow fct制作一个numpy功能: 我们将从将np_d_spiky变为张量流函数开始.张量流tf.py_func(func, inp, Tout, stateful=stateful, name=name)中有一个功能 [doc] 它将任何numpy函数转换为张量流函数，因此我们可以使用它:

Making a numpy fct to a tensorflow fct: We will start by making np_d_spiky into a tensorflow function. There is a function in tensorflow tf.py_func(func, inp, Tout, stateful=stateful, name=name) [doc] which transforms any numpy function to a tensorflow function, so we can use it:

import tensorflow as tf
from tensorflow.python.framework import ops

np_d_spiky_32 = lambda x: np_d_spiky(x).astype(np.float32)


def tf_d_spiky(x,name=None):
    with tf.name_scope(name, "d_spiky", [x]) as name:
        y = tf.py_func(np_d_spiky_32,
                        [x],
                        [tf.float32],
                        name=name,
                        stateful=False)
        return y[0]

tf.py_func作用于张量列表(并返回张量列表)，这就是为什么我们有[x](并返回y[0])的原因. stateful选项是告诉tensorflow函数是否总是为相同的输入提供相同的输出(有状态= False)，在这种情况下，tensorflow可以简单地表示tensorflow图，这是我们的情况，并且在大多数情况下可能都是这种情况.此时要注意的一件事是numpy使用float64但tensorflow使用float32，因此您需要先将函数转换为使用float32，然后才能将其转换为tensorflow函数，否则tensorflow会抱怨.这就是为什么我们需要先制作np_d_spiky_32的原因.

tf.py_func acts on lists of tensors (and returns a list of tensors), that is why we have [x] (and return y[0]). The stateful option is to tell tensorflow whether the function always gives the same output for the same input (stateful = False) in which case tensorflow can simply the tensorflow graph, this is our case and will probably be the case in most situations. One thing to be careful of at this point is that numpy used float64 but tensorflow uses float32 so you need to convert your function to use float32 before you can convert it to a tensorflow function otherwise tensorflow will complain. This is why we need to make np_d_spiky_32 first.

梯度是什么?仅执行上述操作的问题是，即使我们现在有了tf_d_spiky，它是np_d_spiky的张量流版本，我们也无法将其用作如果我们想要激活函数，因为tensorflow不知道如何计算该函数的梯度.

What about the Gradients? The problem with only doing the above is that even though we now have tf_d_spiky which is the tensorflow version of np_d_spiky, we couldn't use it as an activation function if we wanted to because tensorflow doesn't know how to calculate the gradients of that function.

获取渐变的技巧:如上述资源中所述，有一种使用tf.RegisterGradient harpone 复制代码，我们可以修改tf.py_func函数以使其在同时:

Hack to get Gradients: As explained in the sources mentioned above, there is a hack to define gradients of a function using tf.RegisterGradient [doc] and tf.Graph.gradient_override_map [doc]. Copying the code from harpone we can modify the tf.py_func function to make it define the gradient at the same time:

def py_func(func, inp, Tout, stateful=True, name=None, grad=None):

    # Need to generate a unique name to avoid duplicates:
    rnd_name = 'PyFuncGrad' + str(np.random.randint(0, 1E+8))

    tf.RegisterGradient(rnd_name)(grad)  # see _MySquareGrad for grad example
    g = tf.get_default_graph()
    with g.gradient_override_map({"PyFunc": rnd_name}):
        return tf.py_func(func, inp, Tout, stateful=stateful, name=name)

现在我们差不多完成了，唯一的事情是我们需要传递给上述py_func函数的grad函数需要采用一种特殊的形式.它需要接受一个操作，并且需要在操作之前进行先前的渐变，并在操作之后向后传播这些渐变.

Now we are almost done, the only thing is that the grad function we need to pass to the above py_func function needs to take a special form. It needs to take in an operation, and the previous gradients before the operation and propagate the gradients backward after the operation.

梯度函数:因此，对于我们尖刻的激活函数，我们将这样做:

Gradient Function: So for our spiky activation function that is how we would do it:

def spikygrad(op, grad):
    x = op.inputs[0]

    n_gr = tf_d_spiky(x)
    return grad * n_gr  

激活功能只有一个输入，这就是为什么x = op.inputs[0].如果操作有很多输入，我们将需要返回一个元组，每个输入一个梯度.例如，如果操作为a-b，则相对于a的梯度为+1，相对于b的梯度为-1，因此我们将得到return +1*grad,-1*grad.注意，我们需要返回输入的张量流函数，这就是为什么需求tf_d_spiky和np_d_spiky不能起作用，因为它不能作用于张量流张量.或者，我们可以使用张量流函数编写导数:

The activation function has only one input, that is why x = op.inputs[0]. If the operation had many inputs, we would need to return a tuple, one gradient for each input. For example if the operation was a-bthe gradient with respect to a is +1 and with respect to b is -1 so we would have return +1*grad,-1*grad. Notice that we need to return tensorflow functions of the input, that is why need tf_d_spiky, np_d_spiky would not have worked because it cannot act on tensorflow tensors. Alternatively we could have written the derivative using tensorflow functions:

def spikygrad2(op, grad):
    x = op.inputs[0]
    r = tf.mod(x,1)
    n_gr = tf.to_float(tf.less_equal(r, 0.5))
    return grad * n_gr  

将它们组合在一起:现在我们已经拥有了所有的部件，我们可以将它们全部组合在一起:

Combining it all together: Now that we have all the pieces, we can combine them all together:

np_spiky_32 = lambda x: np_spiky(x).astype(np.float32)

def tf_spiky(x, name=None):

    with tf.name_scope(name, "spiky", [x]) as name:
        y = py_func(np_spiky_32,
                        [x],
                        [tf.float32],
                        name=name,
                        grad=spikygrad)  # <-- here's the call to the gradient
        return y[0]

现在我们完成了.我们可以对其进行测试.

And now we are done. And we can test it.

测试:

with tf.Session() as sess:

    x = tf.constant([0.2,0.7,1.2,1.7])
    y = tf_spiky(x)
    tf.initialize_all_variables().run()

    print(x.eval(), y.eval(), tf.gradients(y, [x])[0].eval())

[0.2 0.69999999 1.20000005 1.70000005] [0.2 0. 0.20000005 0.] [1. 0. 1. 0.]

[ 0.2 0.69999999 1.20000005 1.70000005] [ 0.2 0. 0.20000005 0.] [ 1. 0. 1. 0.]

成功！

这篇关于如何在Tensorflow中仅使用Python制作自定义激活功能?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！