从NLTK中的Text.similar()和ContextIndex.similar_words()生成的单词按频率排序? [英] Words generated from Text.similar() and ContextIndex.similar_words() in NLTK sorted by frequency?
问题描述
我正在使用这两个函数来查找相似的单词，并且它们返回不同的列表.我想知道这些功能是否按最频繁到最不频繁的关联排序?
Text.similar() 只是计算单词共享的唯一上下文的数量. /p>
similar_words()似乎包含NLTK 2.0中的错误.请参见 nltk/text.py
def similar_words(self, word, n=20):
scores = defaultdict(int)
for c in self._word_to_contexts[self._key(word)]:
for w in self._context_to_words[c]:
if w != word:
print w, c, self._context_to_words[c][word], self._context_to_words[c][w]
scores[w] += self._context_to_words[c][word] * self._context_to_words[c][w]
return sorted(scores, key=scores.get)[:n]
返回的单词列表应按照相似性得分的降序排列.将return语句替换为:
return sorted(scores, key=scores.get)[::-1][:n]
在similar()中，对similar_words()的调用被注释掉了，这可能是由于此错误所致.
def similar(self, word, num=20):
if '_word_context_index' not in self.__dict__:
print 'Building word-context index...'
self._word_context_index = ContextIndex(self.tokens,
filter=lambda x:x.isalpha(),
key=lambda s:s.lower())
# words = self._word_context_index.similar_words(word, num)
word = word.lower()
wci = self._word_context_index._word_to_contexts
if word in wci.conditions():
contexts = set(wci[word])
fd = FreqDist(w for w in wci.conditions() for c in wci[w]
if c in contexts and not w == word)
words = fd.keys()[:num]
print tokenwrap(words)
else:
print "No matches"
注意:在FreqDist中，与dict不同，keys()返回排序列表.
示例:
import nltk
text = nltk.Text(word.lower() for word in nltk.corpus.brown.words())
text.similar('woman')
similar_words = text._word_context_index.similar_words('woman')
print ' '.join(similar_words)
输出:
man day time year car moment world family house boy child country
job state girl place war way case question # Text.similar()
#man ('a', 'who') 9 39 # output from similar_words(); see following explanation
#girl ('a', 'who') 9 6
#[...]
man number time world fact end year state house way day use part
kind boy matter problem result girl group # ContextIndex.similar_words()
fd(similar()中的频率分布)是每个单词的上下文数量的总和:
fd = [('man', 52), ('day', 30), ('time', 30), ('year', 28), ('car', 24), ('moment', 24), ('world', 23) ...]
对于每个上下文中的每个单词，similar_words()计算频率乘积的总和:
man ('a', 'who') 9 39 # 'a man who' occurs 39 times in text;
# 'a woman who' occurs 9 times
# Similarity score for the context is the product:
# score['man'] = 9 * 39
girl ('a', 'who') 9 6
writer ('a', 'who') 9 4
boy ('a', 'who') 9 3
child ('a', 'who') 9 2
dealer ('a', 'who') 9 2
...
man ('a', 'and') 6 11 # score += 6 * 11
...
man ('a', 'he') 4 6 # score += 4 * 6
...
[49 more occurrences of 'man']
I'm using these two functions to find similar words and they return different lists. I'm wondering if these functions are sorted by most to least frequent association?
ContextIndex.similar_words(word) calculates the similarity score for each word as the sum of the products of frequencies in each context.
Text.similar() simply counts the number of unique contexts the words share.
similar_words() seems to contain a bug in NLTK 2.0. See the definition in nltk/text.py:
def similar_words(self, word, n=20):
scores = defaultdict(int)
for c in self._word_to_contexts[self._key(word)]:
for w in self._context_to_words[c]:
if w != word:
print w, c, self._context_to_words[c][word], self._context_to_words[c][w]
scores[w] += self._context_to_words[c][word] * self._context_to_words[c][w]
return sorted(scores, key=scores.get)[:n]
The returned word list should be sorted in descending order of similarity score. Replace the return statement with:
return sorted(scores, key=scores.get)[::-1][:n]
In similar(), the call to similar_words() is commented out, perhaps due to this bug.
def similar(self, word, num=20):
if '_word_context_index' not in self.__dict__:
print 'Building word-context index...'
self._word_context_index = ContextIndex(self.tokens,
filter=lambda x:x.isalpha(),
key=lambda s:s.lower())
# words = self._word_context_index.similar_words(word, num)
word = word.lower()
wci = self._word_context_index._word_to_contexts
if word in wci.conditions():
contexts = set(wci[word])
fd = FreqDist(w for w in wci.conditions() for c in wci[w]
if c in contexts and not w == word)
words = fd.keys()[:num]
print tokenwrap(words)
else:
print "No matches"
Note: in a FreqDist, unlike a dict, keys() returns a sorted list.
Example:
import nltk
text = nltk.Text(word.lower() for word in nltk.corpus.brown.words())
text.similar('woman')
similar_words = text._word_context_index.similar_words('woman')
print ' '.join(similar_words)
Output:
man day time year car moment world family house boy child country
job state girl place war way case question # Text.similar()
#man ('a', 'who') 9 39 # output from similar_words(); see following explanation
#girl ('a', 'who') 9 6
#[...]
man number time world fact end year state house way day use part
kind boy matter problem result girl group # ContextIndex.similar_words()
fd, the frequency distribution in similar(), is a tally of the number of contexts for each word:
fd = [('man', 52), ('day', 30), ('time', 30), ('year', 28), ('car', 24), ('moment', 24), ('world', 23) ...]
For each word in each context, similar_words() calculates the sum of the product of the frequencies:
man ('a', 'who') 9 39 # 'a man who' occurs 39 times in text;
# 'a woman who' occurs 9 times
# Similarity score for the context is the product:
# score['man'] = 9 * 39
girl ('a', 'who') 9 6
writer ('a', 'who') 9 4
boy ('a', 'who') 9 3
child ('a', 'who') 9 2
dealer ('a', 'who') 9 2
...
man ('a', 'and') 6 11 # score += 6 * 11
...
man ('a', 'he') 4 6 # score += 4 * 6
...
[49 more occurrences of 'man']
这篇关于从NLTK中的Text.similar()和ContextIndex.similar_words()生成的单词按频率排序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
