数字相同的代码段产生截然不同的结果 [英] pieces of numerically identical code produce drastically different results

问题描述

与以下有关:tensorflow 修改 py_func(及其 grad func)中的变量

我用这个函数在 TensorFlow 中定义了我自己的 op 和它的梯度.

I define my own op and its gradient in TensorFlow with this function.

    # define gradient of a python function
def py_func_with_grad(func, inp, Tout, stateful=True, name=None, grad=None): 
    num = []
    for i in range(100):
        num.append(str(np.random.randint(0,10)))
    rnd_name = 'PyFuncGrad' + ''.join(num)
    tf.RegisterGradient(rnd_name)(grad)
    g = tf.get_default_graph()
    with g.gradient_override_map({"PyFunc": rnd_name}):
        return tf.py_func(func, inp, Tout, stateful=stateful, name=name)

我有一个神经网络，其中包含以下代码片段，其中有 5 条数字相同的行(我使用了其中的一条).它们产生截然不同的结果.我想知道是否有人有任何线索.谢谢！！

I have a neural network that contains the following code snippet, where I have 5 numerically identical lines (I use one of them once). They produce drastically different results. I wonder if anybody has any clue. Thanks!!

例如，在 (1) 和 (2) 中，仅仅用 TF 变量 (s_final) 替换 x 就可以产生如此大的不同，这太奇怪了.我想既然它们在数字上相同，那么应该没有任何区别.

For example, it's so weird that in (1) and (2) by merely replacing x with a TF variable (s_final) can make such a difference. I thought since they are numerically the same, there shouldn't be any difference.

s_final 是一个 Tensorflow 不可训练变量.

s_final is a Tensorflow non-trainable variable.

    def _idenity_func(x,s): 
        return s
    def _dummy_grad(op,grad):
        return grad*0,grad*0

    assign_op_s_final_2 = s_final.assign(x)
    with tf.control_dependencies( [assign_op_s_final_2] ):
        x = tf.identity(x)

    x = tf.stop_gradient(x)

    # the three following lines should be numerically identical. since s_final has been assigned the value of x. but...
    # (1) use the following line, the network does not learn AT ALL!!
    x_revised = py_func_with_grad(_idenity_func, [x, s_final], [tf.float32], name=name, grad=lambda op,grad: _dummy_grad(op,grad) )
    # (2) use the following line, the network learns, even if x does not need any gradient (since there is tf.stop_gradient)
    # x_revised = py_func_with_grad(_idenity_func, [x, x], [tf.float32], name=name, grad=lambda op,grad: _dummy_grad(op,grad)) 
    # (3) use the following line, the network learns as well as (2)
    # x_revised = tf.stop_gradient(x) 
    # (4) use the following line, the network learns, but seems not as well as (2)  
    # x_revised = tf.stop_gradient(s_final)
    # (5) use the following line, the network does not learn AT ALL!!
    # x_revised = py_func_with_grad(_idenity_func, [x, tf.stop_gradient(s_final)], [tf.float32], name=name, grad=lambda op,grad: _dummy_grad(op,grad) )

提供了代码(需要 tensorflow 0.12.1.不适用于版本 >=1，因为 HyperNetworks 的实现不支持 tensorflow 版本 >=1):

Code is provided (requires tensorflow 0.12.1. Does not work with version >=1 because the implementation of HyperNetworks does not support tensorflow version >=1):

https://www.dropbox.com/s/58khyqdy3mtnri7/tensorflow_clean_ver01.zip?dl=0

以上几行在我们提供的代码中.更改它们并运行模型以查看差异.如有任何关于代码的问题，请告诉我.

The above lines are in the code we provide. Change them and run the model to see difference. Let me know any question about the code.

您可以将 tensorflow 0.12.1 安装到临时文件夹:

You can install tensorflow 0.12.1 to a temporary folder:

export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.1-cp27-none-linux_x86_64.whl
pip install --target=$HOME/tensorflow_versions/tf-0.12.1  --upgrade $TF_BINARY_URL

然后在您运行提供的代码时添加路径.我使用这种方法在我的计算机上安装了多个版本的 Tensorflow.

Then the path is added when you run the provided code. I use this approach to have multiple versions of Tensorflow on my computer.

推荐答案

我的猜测是赋值并没有真正执行.请注意，您只是在构建图形，还没有执行任何操作(不像 pytorch)...

My guess is that the assign is not really executed. Note that you are just building the graph, nothing is executed yet (not like pytorch)...

这篇关于数字相同的代码段产生截然不同的结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！