nltk:如何获取包含特定单词的双字母 [英] nltk: how to get bigrams containing a specific word

问题描述

我是nltk的新手，并且想获取特定单词(例如"man")的搭配，以便以后我可以按频率过滤它们并按PMI分数对其进行排序.

I am new to nltk, and would like to get the collocates of a specific word (e.g. "man") so that later I would filter them by frequency and sort them by PMI score.

这是我的试验代码，用于检索包含"man"的双字母组，但它返回一个空列表:

Here is my trial code to retrieve the bigrams containing "man", but it returns an empty list:

>>> text = "hello, yesterday I have seen a man walking. On the other side there was another man yelling \"who are you, man?\""
>>> tokens = word_tokenize(text)
>>> finder = BigramCollocationFinder.from_words(tokens, window_size=5)
>>> filter_man = lambda w: "man" not in w
>>> finder.apply_word_filter(filter_man)
>>> finder.ngram_fd.items()
[(('have', 'seen'), 1), ((',', 'yesterday'), 1), (('on', 'the'), 1), (('I', 'have'), 1), (('of', 'another'), 1), (('walking', 'on'), 1), (('seen', 'a'), 1), (('hello', ','), 1), (('man', 'walking'), 1), (('side', 'of'), 1), (('the', 'opposite'), 1), (('a', 'man'), 1), (('opposite', 'side'), 1), (('another', 'man'), 1), (('yesterday', 'I'), 1)]
>>> finder.ngram_fd.items()
[]
>>> 

我在做什么错了?

推荐答案

finder = BigramCollocationFinder.from_words(text.split())
word_filter = lambda w1, w2: "man" not in (w1, w2)
finder.apply_ngram_filter(word_filter)

bigram_measures = nltk.collocations.BigramAssocMeasures()
raw_freq_ranking = finder.nbest(bigram_measures.raw_freq, 10) #top-10

这篇关于nltk:如何获取包含特定单词的双字母的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！