使用 TensorFlow 重新归一化权重矩阵 [英] Renormalize weight matrix using TensorFlow

问题描述

我想为我的 TensorFlow 图中的几个权重矩阵添加一个最大范数约束，ala Torch 的 renorm 方法.

如果任何神经元的权重矩阵的 L2 范数超过 max_norm，我想缩小它的权重，以便它们的 L2 范数正好是 max_norm.

使用 TensorFlow 表达这一点的最佳方式是什么?

解决方案

这是一个可能的实现:

 将 tensorflow 导入为 tfdef maxnorm_regularizer(阈值，轴= 1，名称=maxnorm"，集合=maxnorm"):def maxnorm(权重):裁剪 = tf.clip_by_norm(权重，clip_norm=阈值，轴=轴)clip_weights = tf.assign(weights, clipped, name=name)tf.add_to_collection(集合，clip_weights)return None # 没有正则化损失项返回最大范数

您将如何使用它:

from tensorflow.contrib.layers importfully_connected从 tensorflow.contrib.framework 导入 arg_scope与 arg_scope([完全连接]，weights_regularizer=max_norm_regularizer(1.5)):hidden1 = full_connected(X, 200, scope="hidden1")hidden2 = full_connected(hidden1, 100, scope="hidden2")输出 = 完全连接(hidden2, 5, activation_fn=None, scope="outs")max_norm_ops = tf.get_collection("max_norm")[...]使用 tf.Session() 作为 sess:sess.run(初始化)对于范围内的纪元(n_epochs):对于 load_next_batch() 中的 X_batch、y_batch:sess.run(training_op, feed_dict={X: X_batch, y: y_batch})sess.run(max_norm_ops)

这将创建一个 3 层神经网络，并在每一层(阈值为 1.5)使用最大范数正则化对其进行训练.我刚试过，似乎有效.希望这可以帮助！欢迎提出改进建议.:)

注意事项

此代码基于tf.clip_by_norm():

<预><代码>>>>x = tf.constant([0., 0., 3., 4., 30., 40., 300., 400.], shape=(4, 2))>>>打印(x.eval())[[0.0.][ 3. 4. ][ 30. 40. ][ 300. 400.]]>>>clip_rows = tf.clip_by_norm(x，clip_norm=10，轴=1)>>>打印(clip_rows.eval())[[ 0. 0. ][ 3. 4. ][ 6. 8. ] # 剪掉了！[ 6.00000048 8. ]] # 剪掉了！

如果需要，您还可以剪辑列:

<预><代码>>>>clip_cols = tf.clip_by_norm(x，clip_norm=350，轴=0)>>>打印(clip_cols.eval())[[ 0. 0. ][ 3. 3.48245788][ 30. 34.82457733][300.348.24578857]]# 剪辑！

I'd like to add a max norm constraint to several of the weight matrices in my TensorFlow graph, ala Torch's renorm method.

If the L2 norm of any neuron's weight matrix exceeds max_norm, I'd like to scale its weights down so that their L2 norm is exactly max_norm.

What's the best way to express this using TensorFlow?

解决方案

Here is a possible implementation:

import tensorflow as tf

def maxnorm_regularizer(threshold, axes=1, name="maxnorm", collection="maxnorm"):
    def maxnorm(weights):
        clipped = tf.clip_by_norm(weights, clip_norm=threshold, axes=axes)
        clip_weights = tf.assign(weights, clipped, name=name)
        tf.add_to_collection(collection, clip_weights)
        return None # there is no regularization loss term
    return maxnorm

Here's how you would use it:

from tensorflow.contrib.layers import fully_connected
from tensorflow.contrib.framework import arg_scope

with arg_scope(
        [fully_connected],
        weights_regularizer=max_norm_regularizer(1.5)):
    hidden1 = fully_connected(X, 200, scope="hidden1")
    hidden2 = fully_connected(hidden1, 100, scope="hidden2")
    outputs = fully_connected(hidden2, 5, activation_fn=None, scope="outs")

max_norm_ops = tf.get_collection("max_norm")

[...]

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(n_epochs):
        for X_batch, y_batch in load_next_batch():
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
            sess.run(max_norm_ops)

This creates a 3 layer neural network and trains it with max norm regularization at every layer (with a threshold of 1.5). I just tried it, seems to work. Hope this helps! Suggestions for improvements are welcome. :)

Notes

This code is based on tf.clip_by_norm():

>>> x = tf.constant([0., 0., 3., 4., 30., 40., 300., 400.], shape=(4, 2))
>>> print(x.eval())
[[   0.    0.]
 [   3.    4.]
 [  30.   40.]
 [ 300.  400.]]
>>> clip_rows = tf.clip_by_norm(x, clip_norm=10, axes=1)
>>> print(clip_rows.eval())
[[ 0.          0.        ]
 [ 3.          4.        ]
 [ 6.          8.        ]  # clipped!
 [ 6.00000048  8.        ]] # clipped!

You can also clip columns if you need to:

>>> clip_cols = tf.clip_by_norm(x, clip_norm=350, axes=0)
>>> print(clip_cols.eval())
[[   0.            0.        ]
 [   3.            3.48245788]
 [  30.           34.82457733]
 [ 300.          348.24578857]]
                # clipped!

这篇关于使用 TensorFlow 重新归一化权重矩阵的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！