SpaCy:如何加载 Google 新闻 word2vec 向量? [英] SpaCy: how to load Google news word2vec vectors?
问题描述
我尝试了几种加载谷歌新闻 word2vec 向量的方法(https://code.google.com/archive/p/word2vec/):
I've tried several methods of loading the google news word2vec vectors (https://code.google.com/archive/p/word2vec/):
en_nlp = spacy.load('en',vector=False)
en_nlp.vocab.load_vectors_from_bin_loc('GoogleNews-vectors-negative300.bin')
以上给出:
MemoryError: Error assigning 18446744072820359357 bytes
我也尝试过使用 .gz 压缩向量;或者通过 gensim 加载并保存它们为新格式:
I've also tried with the .gz packed vectors; or by loading and saving them with gensim to a new format:
from gensim.models.word2vec import Word2Vec
model = Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
model.save_word2vec_format('googlenews2.txt')
这个文件然后在每一行包含单词和它们的词向量.我试图加载它们:
This file then contains the words and their word vectors on each line. I tried to load them with:
en_nlp.vocab.load_vectors('googlenews2.txt')
但它返回0".
这样做的正确方法是什么?
What is the correct way to do this?
更新:
我可以将自己创建的文件加载到 spacy 中.我在每一行使用一个带有string 0.0 0.0 ...."的 test.txt 文件.然后用 .bzip2 将这个 txt 压缩到 test.txt.bz2.然后我创建了一个兼容 spacy 的二进制文件:
I can load my own created file into spacy. I use a test.txt file with "string 0.0 0.0 ...." on each line. Then zip this txt with .bzip2 to test.txt.bz2. Then I create a spacy compatible binary file:
spacy.vocab.write_binary_vectors('test.txt.bz2', 'test.bin')
我可以加载到 spacy:
That I can load into spacy:
nlp.vocab.load_vectors_from_bin_loc('test.bin')
这有效!但是,当我对 googlenews2.txt 执行相同的过程时,出现以下错误:
This works! However, when I do the same process for the googlenews2.txt, I get the following error:
lib/python3.6/site-packages/spacy/cfile.pyx in spacy.cfile.CFile.read_into (spacy/cfile.cpp:1279)()
OSError:
推荐答案
对于 spacy 1.x,将 Google 新闻向量加载到 gensim 并转换为新格式(.txt 中的每一行都包含一个向量:string, vec):
For spacy 1.x, load Google news vectors into gensim and convert to a new format (each line in .txt contains a single vector: string, vec):
from gensim.models.word2vec import Word2Vec
from gensim.models import KeyedVectors
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
model.wv.save_word2vec_format('googlenews.txt')
删除.txt的第一行:
Remove the first line of the .txt:
tail -n +2 googlenews.txt > googlenews.new && mv -f googlenews.new googlenews.txt
将txt压缩为.bz2:
Compress the txt as .bz2:
bzip2 googlenews.txt
创建一个 SpaCy 兼容的二进制文件:
Create a SpaCy compatible binary file:
spacy.vocab.write_binary_vectors('googlenews.txt.bz2','googlenews.bin')
将 googlenews.bin 移动到您的 Python 环境的/lib/python/site-packages/spacy/data/en_google-1.0.0/vocab/googlenews.bin.
Move the googlenews.bin to /lib/python/site-packages/spacy/data/en_google-1.0.0/vocab/googlenews.bin of your python environment.
然后加载词向量:
import spacy
nlp = spacy.load('en',vectors='en_google')
或稍后加载它们:
nlp.vocab.load_vectors_from_bin_loc('googlenews.bin')
这篇关于SpaCy:如何加载 Google 新闻 word2vec 向量?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!