如何在 Tensorflow RNN 中构建嵌入层? [英] How to build an embedding layer in Tensorflow RNN?
问题描述
我正在构建一个 RNN LSTM 网络,以根据作者的年龄(二元分类 - 年轻/成人)对文本进行分类.
I'm building an RNN LSTM network to classify texts based on the writers' age (binary classification - young / adult).
似乎网络没有学习,突然开始过度拟合:
Seems like the network does not learn and suddenly starts overfitting:
红色:火车
蓝色:验证
一种可能是数据表示不够好.我只是按照它们的频率对独特的词进行排序并给它们索引.例如:
One possibility could be that the data representation is not good enough. I just sorted the unique words by their frequency and gave them indices. E.g.:
unknown -> 0
the -> 1
a -> 2
. -> 3
to -> 4
所以我试图用词嵌入替换它.我看到了几个例子,但我无法在我的代码中实现它.大多数示例如下所示:
So I'm trying to replace that with word embedding. I saw a couple of examples but I'm not able to implement it in my code. Most of the examples look like this:
embedding = tf.Variable(tf.random_uniform([vocab_size, hidden_size], -1, 1))
inputs = tf.nn.embedding_lookup(embedding, input_data)
这是否意味着我们正在构建一个学习嵌入的层?我认为应该下载一些 Word2Vec 或 Glove 并使用它们.
Does this mean we're building a layer that learns the embedding? I thought that one should download some Word2Vec or Glove and just use that.
无论如何,假设我想构建这个嵌入层...
如果我在代码中使用这两行,我会收到一个错误:
Anyway let's say I want to build this embedding layer...
If I use these 2 lines in my code I get an error:
类型错误:传递给参数索引"的值的数据类型 float32 不在允许值列表中:int32、int64
TypeError: Value passed to parameter 'indices' has DataType float32 not in list of allowed values: int32, int64
所以我想我必须将 input_data
类型更改为 int32
.所以我这样做了(毕竟都是指数),我明白了:
So I guess I have to change the input_data
type to int32
. So I do that (it's all indices after all), and I get this:
类型错误:输入必须是一个序列
TypeError: inputs must be a sequence
我尝试用一个列表包装 inputs
(tf.contrib.rnn.static_rnn
的参数):[inputs]
如 这个答案,但这又产生了另一个错误:
I tried wrapping inputs
(argument to tf.contrib.rnn.static_rnn
) with a list: [inputs]
as suggested in this answer, but that produced another error:
ValueError:输入大小(输入的维度 0)必须可以通过形状推断,但看到值无.
ValueError: Input size (dimension 0 of inputs) must be accessible via shape inference, but saw value None.
<小时>
更新:
在将张量 x
传递给 embedding_lookup
之前,我将其拆开.嵌入后我移动了拆垛.
I was unstacking the tensor x
before passing it to embedding_lookup
. I moved the unstacking after the embedding.
更新代码:
MIN_TOKENS = 10
MAX_TOKENS = 30
x = tf.placeholder("int32", [None, MAX_TOKENS, 1])
y = tf.placeholder("float", [None, N_CLASSES]) # 0.0 / 1.0
...
seqlen = tf.placeholder(tf.int32, [None]) #list of each sequence length*
embedding = tf.Variable(tf.random_uniform([VOCAB_SIZE, HIDDEN_SIZE], -1, 1))
inputs = tf.nn.embedding_lookup(embedding, x) #x is the text after converting to indices
inputs = tf.unstack(inputs, MAX_POST_LENGTH, 1)
outputs, states = tf.contrib.rnn.static_rnn(lstm_cell, inputs, dtype=tf.float32, sequence_length=seqlen) #---> Produces error
*seqlen:我对序列进行了零填充,因此它们都具有相同的列表大小,但由于实际大小不同,我准备了一个描述没有填充的长度的列表.
新错误:
ValueError: 层 basic_lstm_cell_1 的输入 0 与图层:预期 ndim=2,发现 ndim=3.收到完整形状:[无,1, 64]
ValueError: Input 0 of layer basic_lstm_cell_1 is incompatible with the layer: expected ndim=2, found ndim=3. Full shape received: [None, 1, 64]
64 是每个隐藏层的大小.
很明显,我的维度有问题......嵌入后如何使输入适合网络?
It's obvious that I have a problem with the dimensions... How can I make the inputs fit the network after embedding?
推荐答案
来自 tf.nn.static_rnn ,我们可以看到 inputs
参数是:
From the tf.nn.static_rnn , we can see the inputs
arguments to be:
一个长度为 T 的输入列表,每个输入一个形状为 [batch_size, input_size] 的张量
A length T list of inputs, each a Tensor of shape [batch_size, input_size]
所以你的代码应该是这样的:
So your code should be something like:
x = tf.placeholder("int32", [None, MAX_TOKENS])
...
inputs = tf.unstack(inputs, axis=1)
这篇关于如何在 Tensorflow RNN 中构建嵌入层?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!