在Keras-预训练词嵌入中平均句子的词向量 [英] averaging a sentence’s word vectors in Keras- Pre-trained Word Embedding

问题描述

我是Keras的新手.

I am new to Keras.

我的目标是为推文创建用于情感分析的神经网络多分类.

My goal is to create a Neural Network Multi-Classification for Sentiment Analysis for tweets.

我使用了Keras中的Sequential来建立我的模型.

I used Sequential in Keras to build my model.

我想在模型的第一层(特别是gloVe)中使用预训练词嵌入.

I want to use pre-trained word embeddings in the first layer of my model, specifically gloVe.

这是我目前的模特:

model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], input_length=max_length, trainable=False))
model.add(LSTM(100, stateful=False))
model.add(Dense(8, input_dim=4, activation='relu'))
model.add(Dense(3, activation='softmax'))

embedding_matrix由来自文件glove.840B.300d.txt

由于我对神经网络模型的输入是句子(或推文)，并且在参考了一些理论之后，我希望在嵌入层之后的层中，获取推文中的每个单词向量， 平均句子的单词向量.

Since my input to the neural network model is sentences (or tweets), and after consulting some theory, I want for the layer after the Embedding layer, after taking every word vector in the tweet, to average the sentence’s word vectors.

目前我使用的是LSTM，我想用这种平均技术或p-means代替它.我在keras文档中找不到此内容.

Currently what I use is LSTM, I want to replace it with this technique of averaging technique or p-means. I wasn't able to find this in keras documentation.

我不确定这是否是问这个问题的正确地点，但是所有帮助都会感激不尽.

I'm not sure if this is the right place to ask this, but all help will be appreciated.

推荐答案

您可以使用Keras后端的mean函数并将其包装在Lambda层中，以对单词的平均嵌入.

You can use the mean function from Keras' backend and wrap it in a Lambda layer to average the embeddings over the words.

import keras
from keras.layers import Embedding
from keras.models import Sequential
import numpy as np
# Set parameters
vocab_size=1000
max_length=10
# Generate random embedding matrix for sake of illustration
embedding_matrix = np.random.rand(vocab_size,300)

model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], 
input_length=max_length, trainable=False))
# Average the output of the Embedding layer over the word dimension
model.add(keras.layers.Lambda(lambda x: keras.backend.mean(x, axis=1)))

model.summary()

赠予:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_6 (Embedding)      (None, 10, 300)           300000    
_________________________________________________________________
lambda_6 (Lambda)            (None, 300)               0         
=================================================================
Total params: 300,000
Trainable params: 0
Non-trainable params: 300,000

此外，您可以使用Lambda层包装对Keras层中的张量进行运算的任意函数，并将其添加到模型中.如果您使用TensorFlow后端，那么您还可以访问tensorflow操作:

Furthermore, you can use the Lambda layer to wrap arbitrary functions that operate on tensors in a Keras layer and add them to your model. If you are using the TensorFlow backend, you have access to tensorflow ops as well:

import tensorflow as tf    
model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], 
input_length=max_length, trainable=False))
model.add(keras.layers.Lambda(lambda x: tf.reduce_mean(x, axis=1)))
# same model as before

这可以帮助实现更多的自定义平均功能.

This can help to implement more custom averaging functions.

这篇关于在Keras-预训练词嵌入中平均句子的词向量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！