在 keras 中使用带有 LSTM nn 的 Gensim Fasttext 模型 [英] Using Gensim Fasttext model with LSTM nn in keras

问题描述

我已经在非常短的句子(最多 10 个单词)的语料库上使用 Gensim 训练了 fasttext 模型.我知道我的测试集包括不在我的训练语料库中的词，即我的语料库中的一些词就像Oxytocin"一样.Lexitocin"、Ematrophin"、Betaxitocin"

I have trained fasttext model with Gensim over the corpus of very short sentences (up to 10 words). I know that my test set includes words that are not in my train corpus, i.e some of the words in my corpus are like "Oxytocin" "Lexitocin", "Ematrophin",'Betaxitocin"

给定测试集中的一个新词，fasttext 非常清楚地知道如何使用字符级别的 n-gram 生成一个与训练集中其他相似词具有高余弦相似度的向量

given a new word in the test set, fasttext knows pretty well to generate a vector with high cosine-similarity to the other similar words in the train set by using the characters level n-gram

如何将 fasttext 模型合并到 LSTM keras 网络中，同时又不会将 fasttext 模型丢失到词汇表中的向量列表中?因为那样的话，即使 fasttext 做得很好，我也不会处理任何 OOV.

How do i incorporate the fasttext model inside a LSTM keras network without losing the fasttext model to just a list of vectors in the vocab? because then I won't handle any OOV even when fasttext do it well.

有什么想法吗?

推荐答案

这里是将 fasttext 模型合并到 LSTM Keras 网络中的过程

here the procedure to incorporate the fasttext model inside an LSTM Keras network

# define dummy data and precproces them

docs = ['Well done',
        'Good work',
        'Great effort',
        'nice work',
        'Excellent',
        'Weak',
        'Poor effort',
        'not good',
        'poor work',
        'Could have done better']

docs = [d.lower().split() for d in docs]

# train fasttext from gensim api

ft = FastText(size=10, window=2, min_count=1, seed=33)
ft.build_vocab(docs)
ft.train(docs, total_examples=ft.corpus_count, epochs=10)

# prepare text for keras neural network

max_len = 8

tokenizer = tf.keras.preprocessing.text.Tokenizer(lower=True)
tokenizer.fit_on_texts(docs)

sequence_docs = tokenizer.texts_to_sequences(docs)
sequence_docs = tf.keras.preprocessing.sequence.pad_sequences(sequence_docs, maxlen=max_len)

# extract fasttext learned embedding and put them in a numpy array

embedding_matrix_ft = np.random.random((len(tokenizer.word_index) + 1, ft.vector_size))

pas = 0
for word,i in tokenizer.word_index.items():
    
    try:
        embedding_matrix_ft[i] = ft.wv[word]
    except:
        pas+=1

# define a keras model and load the pretrained fasttext weights matrix

inp = Input(shape=(max_len,))
emb = Embedding(len(tokenizer.word_index) + 1, ft.vector_size, 
                weights=[embedding_matrix_ft], trainable=False)(inp)
x = LSTM(32)(emb)
out = Dense(1)(x)

model = Model(inp, out)

model.predict(sequence_docs)

如何处理看不见的文字

unseen_docs = ['asdcs work','good nxsqa zajxa']
unseen_docs = [d.lower().split() for d in unseen_docs]

sequence_unseen_docs = tokenizer.texts_to_sequences(unseen_docs)
sequence_unseen_docs = tf.keras.preprocessing.sequence.pad_sequences(sequence_unseen_docs, maxlen=max_len)

model.predict(sequence_unseen_docs)

这篇关于在 keras 中使用带有 LSTM nn 的 Gensim Fasttext 模型的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！