列表中两个单词之间的余弦相似度 [英] cosine similarity between two words in a list
问题描述
我正在定义一个函数,该函数接受一个单词列表,并返回列表中彼此之间具有非零余弦相似度(以及相似度值)的单词的相关信息.
谁能帮我解决这个问题.我在想,如果我能得到一个预先计算好的 word2vec 矢量文件,那会很有帮助,但互联网上没有.
你可以定义这两个函数
def word2vec(word):从集合导入计数器从数学导入 sqrt# 统计word中的字符cw = 计数器(字)# 预先计算一组不同的字符sw = 设置(顺时针)# 预先计算词向量的长度"lw = sqrt(sum(c*c for c in cw.values()))# 返回一个元组返回 cw、sw、lwdef cosdis(v1, v2):# 这两个词的共同点是什么?common = v1[1].intersection(v2[1])# 根据余弦距离的定义,我们有return sum(v1[0][ch]*v2[0][ch] for ch in common)/v1[2]/v2[2]
并在本例中使用它们
<预><代码>>>>a = 'safasfeqefscwaeeafweeaeawaw'>>>b = 'tsafdstrdfadsdfdswdfafdwaed'>>>c = 'optykop;lvhopijresokpghwji7'>>>>>>va = word2vec(a)>>>vb = word2vec(b)>>>vc = word2vec(c)>>>>>>打印 cosdis(va,vb)0.551843662321>>>打印 cosdis(vb,vc)0.113746579656>>>打印 cosdis(vc,va)0.153494378078顺便说一句,您在标签中提到的 word2vec
这是一项完全不同的业务,这需要我们中的一个人花费大量时间和精力来研究它,然后猜猜看,我不是那个人......
I am defining a function which takes a list of words and returns information about the words in the list that have non-zero, cosine similarity between each other (along with the similarity value).
Can anyone help me out with this. I was thinking if I can get a precomputed word2vec vector file then it would be very helpful,but there is none on the internet.
You could define these two functions
def word2vec(word):
from collections import Counter
from math import sqrt
# count the characters in word
cw = Counter(word)
# precomputes a set of the different characters
sw = set(cw)
# precomputes the "length" of the word vector
lw = sqrt(sum(c*c for c in cw.values()))
# return a tuple
return cw, sw, lw
def cosdis(v1, v2):
# which characters are common to the two words?
common = v1[1].intersection(v2[1])
# by definition of cosine distance we have
return sum(v1[0][ch]*v2[0][ch] for ch in common)/v1[2]/v2[2]
and use them as in this example
>>> a = 'safasfeqefscwaeeafweeaeawaw'
>>> b = 'tsafdstrdfadsdfdswdfafdwaed'
>>> c = 'optykop;lvhopijresokpghwji7'
>>>
>>> va = word2vec(a)
>>> vb = word2vec(b)
>>> vc = word2vec(c)
>>>
>>> print cosdis(va,vb)
0.551843662321
>>> print cosdis(vb,vc)
0.113746579656
>>> print cosdis(vc,va)
0.153494378078
BTW, the word2vec
that you mention in a tag is quite a different business, that requires that one of us take a great deal of time and commitment for studying it and guess what, I'm not that one...
这篇关于列表中两个单词之间的余弦相似度的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!