SpaCy:如何加载Google新闻word2vec向量? [英] SpaCy: how to load Google news word2vec vectors?

问题描述

我尝试了几种加载Google新闻word2vec向量的方法( https://code .google.com/archive/p/word2vec/):

I've tried several methods of loading the google news word2vec vectors (https://code.google.com/archive/p/word2vec/):

en_nlp = spacy.load('en',vector=False)
en_nlp.vocab.load_vectors_from_bin_loc('GoogleNews-vectors-negative300.bin')

上面给出了:

MemoryError: Error assigning 18446744072820359357 bytes

我也尝试使用.gz压缩向量；或通过将它们与gensim一起加载并保存为新格式:

I've also tried with the .gz packed vectors; or by loading and saving them with gensim to a new format:

from gensim.models.word2vec import Word2Vec
model = Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
model.save_word2vec_format('googlenews2.txt')

然后，此文件在每行上包含单词及其单词向量. 我尝试将它们加载:

This file then contains the words and their word vectors on each line. I tried to load them with:

en_nlp.vocab.load_vectors('googlenews2.txt')

但它返回"0".

正确的方法是什么?

更新:

我可以将自己创建的文件加载到spacy中. 我在每行上使用带有字符串0.0 0.0 ...."的test.txt文件.然后使用.bzip2将这个txt压缩到test.txt.bz2. 然后创建一个兼容spacy的二进制文件:

I can load my own created file into spacy. I use a test.txt file with "string 0.0 0.0 ...." on each line. Then zip this txt with .bzip2 to test.txt.bz2. Then I create a spacy compatible binary file:

spacy.vocab.write_binary_vectors('test.txt.bz2', 'test.bin')

我可以装上spacy:

That I can load into spacy:

nlp.vocab.load_vectors_from_bin_loc('test.bin')

这有效！ 但是，当我对googlenews2.txt执行相同的过程时，出现以下错误:

This works! However, when I do the same process for the googlenews2.txt, I get the following error:

lib/python3.6/site-packages/spacy/cfile.pyx in spacy.cfile.CFile.read_into (spacy/cfile.cpp:1279)()

OSError: 

推荐答案

对于spacy 1.x，请将Google新闻矢量加载到gensim中并转换为新格式(.txt中的每一行都包含单个矢量:string，vec) :

For spacy 1.x, load Google news vectors into gensim and convert to a new format (each line in .txt contains a single vector: string, vec):

from gensim.models.word2vec import Word2Vec
from gensim.models import KeyedVectors
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
model.wv.save_word2vec_format('googlenews.txt')

删除.txt的第一行:

Remove the first line of the .txt:

tail -n +2 googlenews.txt > googlenews.new && mv -f googlenews.new googlenews.txt

将txt压缩为.bz2:

Compress the txt as .bz2:

bzip2 googlenews.txt

创建与SpaCy兼容的二进制文件:

Create a SpaCy compatible binary file:

spacy.vocab.write_binary_vectors('googlenews.txt.bz2','googlenews.bin')

将googlenews.bin移至您的python环境的/lib/python/site-packages/spacy/data/en_google-1.0.0/vocab/googlenews.bin.

Move the googlenews.bin to /lib/python/site-packages/spacy/data/en_google-1.0.0/vocab/googlenews.bin of your python environment.

然后加载单词向量:

import spacy
nlp = spacy.load('en',vectors='en_google')

或稍后加载它们:

nlp.vocab.load_vectors_from_bin_loc('googlenews.bin')

这篇关于SpaCy:如何加载Google新闻word2vec向量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！