使用SparseTensor作为可训练变量? [英] Using SparseTensor as a trainable variable?

问题描述

我正在尝试使用 SparseTensor 来表示完全连接层中的权重变量.
但是，似乎TensorFlow 0.8不允许将SparseTensor用作tf.Variable.
有什么办法可以解决这个问题?

I'm trying to use SparseTensor to represent weight variables in a fully-connected layer.
However, it seems that TensorFlow 0.8 doesn't allow to use SparseTensor as tf.Variable.
Is there any way to go around this?

我尝试过

import tensorflow as tf

a = tf.constant(1)
b = tf.SparseTensor([[0,0]],[1],[1,1])

print a.__class__  # shows <class 'tensorflow.python.framework.ops.Tensor'>
print b.__class__  # shows <class 'tensorflow.python.framework.ops.SparseTensor'>

tf.Variable(a)     # Variable is declared correctly
tf.Variable(b)     # Fail

顺便说一句，我使用SparseTensor的最终目的是永久屏蔽某些密集形式的连接.因此，在计算和应用渐变时会忽略这些修剪的连接.

By the way, my ultimate goal of using SparseTensor is to permanently mask some of connections in dense form. Thus, these pruned connections are ignored while calculating and applying gradients.

在我当前的MLP实现中，SparseTensor及其稀疏形式的 matmul ops成功报告了推理输出.但是，使用SparseTensor声明的权重不会随着训练步骤的进行而受到训练.

In my current implementation of MLP, SparseTensor and its sparse form of matmul ops successfully reports inference outputs. However, the weights declared using SparseTensor aren't trained as training steps go.

推荐答案

作为解决问题的方法，您可以为稀疏张量的值提供tf.Variable(直到Tensorflow v0.8).在这种情况下，必须预先定义稀疏结构，但是权重仍然可以训练.

As a workaround to your problem, you can provide a tf.Variable (until Tensorflow v0.8) for the values of a sparse tensor. The sparsity structure has to be pre-defined in that case, the weights however remain trainable.

weights = tf.Variable(<initial-value>)
sparse_var = tf.SparseTensor(<indices>, weights, <shape>)  # v0.8
sparse_var = tf.SparseTensor(<indices>, tf.identity(weights), <shape>)  # v0.9

这篇关于使用SparseTensor作为可训练变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！