如何使用gensim从语料库中提取短语 [英] How to extract phrases from corpus using gensim

问题描述

为了预处理语料库，我打算从语料库中提取常用短语，为此，我尝试在gensim中使用 Phrases 模型，在下面的代码中进行了尝试，但未获得期望的输出结果.

For preprocessing the corpus I was planing to extarct common phrases from the corpus, for this I tried using Phrases model in gensim, I tried below code but it's not giving me desired output.

我的代码

from gensim.models import Phrases
documents = ["the mayor of new york was there", "machine learning can be useful sometimes"]

sentence_stream = [doc.split(" ") for doc in documents]
bigram = Phrases(sentence_stream)
sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])

输出

[u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']

但这应该是

[u'the', u'mayor', u'of', u'new_york', u'was', u'there']

但是当我尝试打印火车数据的词汇时，我可以看到bigram，但是它不能与测试数据一起使用，这是我哪里出错了?

But when I tried to print vocab of train data, I can see bigram, but its not working with test data, where I am going wrong?

print bigram.vocab

defaultdict(<type 'int'>, {'useful': 1, 'was_there': 1, 'learning_can': 1, 'learning': 1, 'of_new': 1, 'can_be': 1, 'mayor': 1, 'there': 1, 'machine': 1, 'new': 1, 'was': 1, 'useful_sometimes': 1, 'be': 1, 'mayor_of': 1, 'york_was': 1, 'york': 1, 'machine_learning': 1, 'the_mayor': 1, 'new_york': 1, 'of': 1, 'sometimes': 1, 'can': 1, 'be_useful': 1, 'the': 1}) 

推荐答案

我找到了解决问题的方法，有两个我不关心的参数应该传递给 Phrases()模型，那是

I got the solution for the problem , There was two parameters I didn't take care of it which should be passed to Phrases() model, those are

1. min_count 忽略所有收集的总字数少于此数的单词和双字母词. 默认情况下，该值为5

1. min_count ignore all words and bigrams with total collected count lower than this. Bydefault it value is 5

阈值代表形成短语的阈值(越高，短语越少).如果(cnt(a，b)-min_count)* N/(cnt(a)* cnt(b))>阈值，则接受单词a和b的短语，其中N是总词汇量. 默认情况下，该值为10.0

threshold represents a threshold for forming the phrases (higher means fewer phrases). A phrase of words a and b is accepted if (cnt(a, b) - min_count) * N / (cnt(a) * cnt(b)) > threshold, where N is the total vocabulary size. Bydefault it value is 10.0

在上面的训练数据中有两个语句时，阈值为 0 ，因此我更改了训练数据集并添加了这两个参数.

With my above train data with two statements, threshold value was 0, so I change train datasets and add those two parameters.

我的新代码

from gensim.models import Phrases
documents = ["the mayor of new york was there", "machine learning can be useful sometimes","new york mayor was present"]

sentence_stream = [doc.split(" ") for doc in documents]
bigram = Phrases(sentence_stream, min_count=1, threshold=2)
sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])

输出

[u'the', u'mayor', u'of', u'new_york', u'was', u'there']

Gensim真的很棒:)

Gensim is really awesome :)

这篇关于如何使用gensim从语料库中提取短语的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！