如何在我自己的语料库上训练GloVe算法 [英] How to Train GloVe algorithm on my own corpus

问题描述

我试图遵循此.
但是我浪费了很多时间，最终却毫无用处.
我只想在自己的语料库(〜900Mb corpus.txt文件)上训练GloVe模型. 我下载了上面链接中提供的文件，并使用cygwin对其进行了编译(在编辑demo.sh文件并将其更改为VOCAB_FILE=corpus.txt之后.我应该保持CORPUS=text8不变吗?) 输出为:

I tried to follow this.
But some how I wasted a lot of time ending up with nothing useful.
I just want to train a GloVe model on my own corpus (~900Mb corpus.txt file). I downloaded the files provided in the link above and compiled it using cygwin (after editing the demo.sh file and changed it to VOCAB_FILE=corpus.txt . should I leave CORPUS=text8 unchanged?) the output was:

1. cooccurrence.bin
2. cooccurrence.shuf.bin
3. text8
4. corpus.txt
5. vectors.txt

如何使用这些文件将其作为GloVe模型在python上加载?

How can I used those files to load it as a GloVe model on python?

推荐答案

您可以使用GloVe库来实现:

You can do it using GloVe library:

安装:pip install Gloves_python

Install it: pip install glove_python

然后:

from glove import Corpus, Glove

#Creating a corpus object
corpus = Corpus() 

#Training the corpus to generate the co occurence matrix which is used in GloVe
corpus.fit(lines, window=10)

glove = Glove(no_components=5, learning_rate=0.05) 
glove.fit(corpus.matrix, epochs=30, no_threads=4, verbose=True)
glove.add_dictionary(corpus.dictionary)
glove.save('glove.model')

参考:使用手套进行单词矢量化

这篇关于如何在我自己的语料库上训练GloVe算法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！