Gensim Word2Vec 从预训练模型中选择次要的词向量集 [英] Gensim Word2Vec select minor set of word vectors from pretrained model

问题描述

我在 gensim 中有一个大型的预训练 Word2Vec 模型，我想从中使用预训练的词向量作为 Keras 模型中的嵌入层.

问题是嵌入的大小很大，我不需要大部分词向量(因为我知道哪些词可以作为输入出现).所以我想去掉它们以减小嵌入层的大小.

有没有办法根据单词的白名单只保留所需的词向量(包括相应的索引！)?

解决方案

感谢 这个答案(我已经改了代码一点点使它更好).您可以使用此代码来解决您的问题.

我们在 restricted_word_set 中有我们所有的次要词集(它可以是列表或集合)并且 w2v 是我们的模型，所以这里是函数:

将 numpy 导入为 np定义限制w2v(w2v，restricted_word_set):new_vectors = []new_vocab = {}new_index2entity = []new_vectors_norm = []对于我在范围内(len(w2v.vocab)):字 = w2v.index2entity[i]vec = w2v.vectors[i]vocab = w2v.vocab[词]vec_norm = w2v.vectors_norm[i]如果字在受限制的字集:vocab.index = len(new_index2entity)new_index2entity.append(word)new_vocab[词] = 词汇new_vectors.append(vec)new_vectors_norm.append(vec_norm)w2v.vocab = new_vocabw2v.vectors = np.array(new_vectors)w2v.index2entity = np.array(new_index2entity)w2v.index2word = np.array(new_index2entity)w2v.vectors_norm = np.array(new_vectors_norm)

<块引用>

警告:当您第一次创建模型时，vectors_norm == None 所以如果你在那里使用这个函数，你会得到一个错误.vectors_norm第一次使用后将获得 numpy.ndarray 类型的值.所以在使用该函数之前，请尝试使用 most_similar("cat") 之类的东西vectors_norm 不等于 None.

它根据 Word2VecKeyedVectors.

用法:

w2v = KeyedVectors.load_word2vec_format("GoogleNews-vectors-negative300.bin.gz", binary=True)w2v.most_similar("啤酒")

<块引用>

[('啤酒', 0.8409687876701355),
('啤酒', 0.7733745574951172),
('啤酒', 0.71753990650177),
('饮料', 0.668931245803833),
('贮藏啤酒', 0.6570086479187012),
('Yuengling_Lager', 0.655455470085144),
('微酿', 0.6534324884414673),
('Brooklyn_Lager', 0.6501551866531372),
('suds', 0.6497018337249756),
('brewed_beer', 0.6490240097045898)]

restricted_word_set = {"beer", "wine", "computer", "python", "bash", "lagers"}限制w2v(w2v，restricted_word_set)w2v.most_similar("啤酒")

<块引用>

[('拉格啤酒', 0.6570085287094116),
('酒', 0.6217695474624634),
('bash', 0.20583480596542358),
('计算机', 0.06677375733852386),
('蟒蛇', 0.005948573350906372)]

它也可以用来删除一些单词.

I have a large pretrained Word2Vec model in gensim from which I want to use the pretrained word vectors for an embedding layer in my Keras model.

The problem is that the embedding size is enormous and I don't need most of the word vectors (because I know which words can occure as Input). So I want to get rid of them to reduce the size of my embedding layer.

Is there a way to just keep desired wordvectors (including the coresponding indices!), based on a whitelist of words?

解决方案

Thanks to this answer (I've changed the code a little bit to make it better). you can use this code for solving your problem.

we have all our minor set of words in restricted_word_set(it can be either list or set) and w2v is our model, so here is the function:

import numpy as np

def restrict_w2v(w2v, restricted_word_set):
    new_vectors = []
    new_vocab = {}
    new_index2entity = []
    new_vectors_norm = []

    for i in range(len(w2v.vocab)):
        word = w2v.index2entity[i]
        vec = w2v.vectors[i]
        vocab = w2v.vocab[word]
        vec_norm = w2v.vectors_norm[i]
        if word in restricted_word_set:
            vocab.index = len(new_index2entity)
            new_index2entity.append(word)
            new_vocab[word] = vocab
            new_vectors.append(vec)
            new_vectors_norm.append(vec_norm)

    w2v.vocab = new_vocab
    w2v.vectors = np.array(new_vectors)
    w2v.index2entity = np.array(new_index2entity)
    w2v.index2word = np.array(new_index2entity)
    w2v.vectors_norm = np.array(new_vectors_norm)

WARNING: when you first create the model the vectors_norm == None so you will get an error if you use this function there. vectors_norm will get a value of the type numpy.ndarray after the first use. so before using the function try something like most_similar("cat") so that vectors_norm not be equal to None.

It rewrites all of the variables which are related to the words based on the Word2VecKeyedVectors.

Usage:

w2v = KeyedVectors.load_word2vec_format("GoogleNews-vectors-negative300.bin.gz", binary=True)
w2v.most_similar("beer")

[('beers', 0.8409687876701355),
('lager', 0.7733745574951172),
('Beer', 0.71753990650177),
('drinks', 0.668931245803833),
('lagers', 0.6570086479187012),
('Yuengling_Lager', 0.655455470085144),
('microbrew', 0.6534324884414673),
('Brooklyn_Lager', 0.6501551866531372),
('suds', 0.6497018337249756),
('brewed_beer', 0.6490240097045898)]

restricted_word_set = {"beer", "wine", "computer", "python", "bash", "lagers"}
restrict_w2v(w2v, restricted_word_set)
w2v.most_similar("beer")

[('lagers', 0.6570085287094116),
('wine', 0.6217695474624634),
('bash', 0.20583480596542358),
('computer', 0.06677375733852386),
('python', 0.005948573350906372)]

it can be used for removing some words either.

这篇关于Gensim Word2Vec 从预训练模型中选择次要的词向量集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！