tensorflow梯度-获取所有nan值 [英] tensorflow gradient - getting all nan values

问题描述

我将python 3与anaconda结合使用，将tensorflow 1.12与热切的eval结合使用.

I am using python 3 with anaconda, and tensorflow 1.12 with eager eval.

我正在使用它为暹罗网络创建三重态损失函数，并且需要计算不同数据样本之间的距离.

I am using it to create a triplet loss function for a siamese network, and need to calculate distance between different data samples.

我创建了一个函数来创建距离计算，但是无论我做什么，当我尝试计算相对于网络输出的坡度时，它总是给我所有的nan坡度.

I created a function in order to create the distance calculation, but no matter what I do, when I try to calculate it's gradient with respect to the networks output, It keeps giving me all nan gradient.

这是代码:

def matrix_row_wise_norm(matrix):
    import tensorflow as tf
    tensor = tf.expand_dims(matrix, -1)

    tensor = tf.transpose(tensor, [0, 2, 1]) - tf.transpose(tensor, [2, 0, 1])
    norm = tf.norm(tensor, axis=2)
    return norm

在损失函数中，我正在使用

In the loss function I am using

def loss(y_true, p_pred):
    with tf.GradientTape() as t:
    t.watch(y_pred)
        distance_matrix = matrix_row_wise_norm(y_pred)
        grad = t.gradient(distance_matrix, y_pred)

毕业的都是nan. 我检查了y_pred是由合法值组成的-并且确实如此. 我试图相对于自身创建一个y_pred * 2的渐变，并获得了合法的渐变值.

And the grad is all nans. I checked that y_pred is made of legit values - and it does. I tried to create a gradient of y_pred * 2 with respect to itself and got legitimate gradient values.

我在这里想念什么?距离矩阵的创建中的索引是否有问题?

What am I missing here? Is the indexing in the creation of the distance matrix problematic?

y_pred和loss的dtype均为tf.float32

the dtype of both y_pred and loss is tf.float32

在tf中找到了打开错误报告-这可能是问题吗?

edit: found an open bug report in tf - could this be the issue?

当我将范数轴更改为0或1时，我得到的是合法值，而nan没有任何意义.我用axis=2使用norm的操作是矩阵中行对之间的成对距离，我怀疑这可能与行与自身之间的0距离有关，所以我用min的min值裁剪了值1e-7没有任何运气.

When I change the norm axis to 0 or 1, I am getting legitimate values and nothing goes to nan. The operation I am getting using norm with axis=2 is the pairwise distance between the pairs of rows in the matrix, I suspected this might have something to do with 0 distance between a row to itself, so I clipped the values with min value of 1e-7 without any luck.

谢谢

推荐答案

似乎tf.norm存在数字不稳定性，如此处

Seems that tf.norm suffers from numeric instability as explained here

他们还建议使用l2范数，该范数更加数字稳定，因此我尝试了该方法，由于0梯度，还获得了nan值.因此，我将它们与梯度修剪一起使用，到目前为止，损失函数正在起作用并且设法收敛.

They also suggest using l2 norm that is more numeric stable, So I tried that, also getting nan values, thanks to 0 gradients. So I used those together with gradient clipping, so far so good, the loss function is working and manages to converge.

def last_attempt(y_true, y_pred):
    import tensorflow as tf
    import numpy as np

    loss = tf.zeros(1)

    for i in range(y_pred.shape[0]):
        dist = tf.gather(y_pred, [i], axis=0)
        y = y_true.numpy().squeeze()
        norm = tf.map_fn(tf.nn.l2_loss, dist-y_pred)

        d = norm.numpy()
        d[np.where(y != y[i])] = 0.0
        max_pos = tf.gather(norm, np.argmax(d))

        d = norm.numpy()
        d[np.where(y == y[i])] = np.inf
        min_neg = tf.gather(norm, np.argmin(d))

        loss += tf.clip_by_value(max_pos - min_neg + tf.constant(1, dtype=tf.float32),
                                 1e-8, 1e1)

    return loss

有很多空间可以优化该功能，这里是对我的其他 SO的引用问题-对此进行研究.

There is much room for optimizing that function, here is a reference to my other SO question - working on that.

这篇关于tensorflow梯度-获取所有nan值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！