使用 SparseTensor 作为可训练变量? [英] Using SparseTensor as a trainable variable?

问题描述

我正在尝试使用 SparseTensor 来表示全连接层中的权重变量.
但是，TensorFlow 0.8 似乎不允许使用 SparseTensor 作为 tf.Variable.
有什么办法可以解决这个问题吗?

I'm trying to use SparseTensor to represent weight variables in a fully-connected layer.
However, it seems that TensorFlow 0.8 doesn't allow to use SparseTensor as tf.Variable.
Is there any way to go around this?

我试过了

import tensorflow as tf

a = tf.constant(1)
b = tf.SparseTensor([[0,0]],[1],[1,1])

print a.__class__  # shows <class 'tensorflow.python.framework.ops.Tensor'>
print b.__class__  # shows <class 'tensorflow.python.framework.ops.SparseTensor'>

tf.Variable(a)     # Variable is declared correctly
tf.Variable(b)     # Fail

顺便说一下，我使用 SparseTensor 的最终目标是永久屏蔽一些密集形式的连接.因此，这些修剪后的连接在计算和应用梯度时被忽略.

By the way, my ultimate goal of using SparseTensor is to permanently mask some of connections in dense form. Thus, these pruned connections are ignored while calculating and applying gradients.

在我当前的 MLP 实现中，SparseTensor 及其稀疏形式的 matmul 操作成功地报告了推理输出.但是，使用 SparseTensor 声明的权重不会随着训练步骤的进行而训练.

In my current implementation of MLP, SparseTensor and its sparse form of matmul ops successfully reports inference outputs. However, the weights declared using SparseTensor aren't trained as training steps go.

推荐答案

作为解决您问题的方法，您可以提供一个 tf.Variable(直到 Tensorflow v0.8) 用于稀疏张量的值.在这种情况下，必须预先定义稀疏结构，但权重仍然是可训练的.

As a workaround to your problem, you can provide a tf.Variable (until Tensorflow v0.8) for the values of a sparse tensor. The sparsity structure has to be pre-defined in that case, the weights however remain trainable.

weights = tf.Variable(<initial-value>)
sparse_var = tf.SparseTensor(<indices>, weights, <shape>)  # v0.8
sparse_var = tf.SparseTensor(<indices>, tf.identity(weights), <shape>)  # v0.9

这篇关于使用 SparseTensor 作为可训练变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！