在 Keras 中对句子的词向量求平均 - 预训练词嵌入 [英] averaging a sentence’s word vectors in Keras- Pre-trained Word Embedding

问题描述

我是 Keras 的新手.

I am new to Keras.

我的目标是为推文创建一个用于情感分析的神经网络多分类.

My goal is to create a Neural Network Multi-Classification for Sentiment Analysis for tweets.

我在 Keras 中使用了 Sequential 来构建我的模型.

I used Sequential in Keras to build my model.

我想在模型的第一层使用预训练词嵌入，特别是 gloVe.

I want to use pre-trained word embeddings in the first layer of my model, specifically gloVe.

这是我目前的模型:

model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], input_length=max_length, trainable=False))
model.add(LSTM(100, stateful=False))
model.add(Dense(8, input_dim=4, activation='relu'))
model.add(Dense(3, activation='softmax'))

embedding_matrix 由来自文件 glove.840B.300d.txt

由于我对神经网络模型的输入是句子(或推文)，并且在参考了一些理论之后，我想要嵌入层之后的层，在提取推文中的每个词向量之后，平均句子的词向量.

Since my input to the neural network model is sentences (or tweets), and after consulting some theory, I want for the layer after the Embedding layer, after taking every word vector in the tweet, to average the sentence’s word vectors.

目前我使用的是LSTM，我想用这种平均技术或p-means来代替它.我无法在 keras 文档中找到它.

Currently what I use is LSTM, I want to replace it with this technique of averaging technique or p-means. I wasn't able to find this in keras documentation.

我不确定这是问这个问题的合适地方，但我们将不胜感激.

I'm not sure if this is the right place to ask this, but all help will be appreciated.

推荐答案

您可以使用 Keras 后端的 mean 函数，并将其包裹在 Lambda 层中以求平均值词的嵌入.

You can use the mean function from Keras' backend and wrap it in a Lambda layer to average the embeddings over the words.

import keras
from keras.layers import Embedding
from keras.models import Sequential
import numpy as np
# Set parameters
vocab_size=1000
max_length=10
# Generate random embedding matrix for sake of illustration
embedding_matrix = np.random.rand(vocab_size,300)

model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], 
input_length=max_length, trainable=False))
# Average the output of the Embedding layer over the word dimension
model.add(keras.layers.Lambda(lambda x: keras.backend.mean(x, axis=1)))

model.summary()

给出:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_6 (Embedding)      (None, 10, 300)           300000    
_________________________________________________________________
lambda_6 (Lambda)            (None, 300)               0         
=================================================================
Total params: 300,000
Trainable params: 0
Non-trainable params: 300,000

此外，您可以使用 Lambda 层来包装在 Keras 层中对张量进行操作的任意函数，并将它们添加到您的模型中.如果您使用的是 TensorFlow 后端，您也可以访问 tensorflow 操作:

Furthermore, you can use the Lambda layer to wrap arbitrary functions that operate on tensors in a Keras layer and add them to your model. If you are using the TensorFlow backend, you have access to tensorflow ops as well:

import tensorflow as tf    
model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], 
input_length=max_length, trainable=False))
model.add(keras.layers.Lambda(lambda x: tf.reduce_mean(x, axis=1)))
# same model as before

这有助于实现更多的自定义平均函数.

This can help to implement more custom averaging functions.

这篇关于在 Keras 中对句子的词向量求平均 - 预训练词嵌入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！