使用Keras Tensorflow 2.0获取渐变 [英] Get Gradients with Keras Tensorflow 2.0

问题描述

我想在张量板上跟踪渐变. 但是，由于会话运行语句不再重要，并且 tf.keras.callbacks.TensorBoard 的 write_grads 参数已已被描述，因此我想了解在使用 Keras 或 tensorflow 2.0 进行训练时如何跟踪渐变.

I would like to keep track of the gradients over tensorboard. However, since session run statements are not a thing anymore and the write_grads argument of tf.keras.callbacks.TensorBoard is depricated, I would like to know how to keep track of gradients during training with Keras or tensorflow 2.0.

我当前的方法是为此目的创建一个新的回调类，但是没有成功.也许其他人知道如何完成这种高级工作.

My current approach is to create a new callback class for this purpose, but without success. Maybe someone else knows how to accomplish this kind of advanced stuff.

为测试而创建的代码如下所示，但是会遇到错误，而与将渐变值打印到控制台或张量板无关.

The code created for testing is shown below, but runs into errors independently of printing a gradient value to console or tensorboard.

import tensorflow as tf
from tensorflow.python.keras import backend as K

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu', name='dense128'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax', name='dense10')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])


class GradientCallback(tf.keras.callbacks.Callback):
    console = True

    def on_epoch_end(self, epoch, logs=None):
        weights = [w for w in self.model.trainable_weights if 'dense' in w.name and 'bias' in w.name]
        loss = self.model.total_loss
        optimizer = self.model.optimizer
        gradients = optimizer.get_gradients(loss, weights)
        for t in gradients:
            if self.console:
                print('Tensor: {}'.format(t.name))
                print('{}\n'.format(K.get_value(t)[:10]))
            else:
                tf.summary.histogram(t.name, data=t)


file_writer = tf.summary.create_file_writer("./metrics")
file_writer.set_as_default()

# write_grads has been removed
tensorboard_cb = tf.keras.callbacks.TensorBoard(histogram_freq=1, write_grads=True)
gradient_cb = GradientCallback()

model.fit(x_train, y_train, epochs=5, callbacks=[gradient_cb, tensorboard_cb])

• 将偏差梯度设置为控制台(控制台参数= True) 导致: AttributeError:张量"对象没有属性"numpy"
• 写入张量板(控制台参数= False)将创建: TypeError:不允许将tf.Tensor用作Python bool.使用if t is not None:而不是if t:来测试是否定义了张量，并使用TensorFlow ops(例如tf.cond)执行以 张量的值.
• Priniting bias gradients to console (console parameter = True) leads to: AttributeError: 'Tensor' object has no attribute 'numpy'
• Writing to tensorboard (console parameter = False) creates: TypeError: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None: instead of if t: to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.

推荐答案

要计算权重的损失梯度，请使用

To compute the gradients of the loss against the weights, use

with tf.GradientTape() as tape:
    loss = model(model.trainable_weights)

tape.gradient(loss, model.trainable_weights)

(据说效果不佳)记录在 GradientTape .

This is (arguably poorly) documented on GradientTape.

我们不需要tape.watch变量，因为默认情况下会监视可训练的参数.

We do not need to tape.watch the variable because trainable parameters are watched by default.

作为函数，可以写为

def gradient(model, x):
    x_tensor = tf.convert_to_tensor(x, dtype=tf.float32)
    with tf.GradientTape() as t:
        t.watch(x_tensor)
        loss = model(x_tensor)
    return t.gradient(loss, x_tensor).numpy()

这篇关于使用Keras Tensorflow 2.0获取渐变的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！