仅使用numpy和pandas计算转换矩阵中每个单词的频率 [英] Calculating the frequency of each word in the transition matrix, using numpy and pandas only

问题描述

我试图仅使用numpy和pandas来计算转换矩阵中每个单词的频率.

I am trying to calculate the frequency of each word in the transition matrix, using numpy and pandas only.

我有一个字符串

star_wars = [('darth', 'leia'), ('luke', 'han'), ('chewbacca', 'luke'), 
         ('chewbacca', 'obi'), ('chewbacca', 'luke'), ('leia', 'luke')]

我使用现在，我正在尝试使用

Now I am trying to convert these values of words into probabilities, using this question:

使用交叉表可用于初始数据帧，但仅给我配对

Using a crosstab works for the initial dataframe, but gives me pairs only

pd.crosstab(pd.Series(star_wars[1:]),
        pd.Series(star_wars[:-1]), normalize = 1)

输出错误，这也不适用于我创建的矩阵，只是一个例子:

Output is wrong and this also does not work for my created matrix, just an example:

col_0   (chewbacca, luke)   (chewbacca, obi)    (darth, leia)   (luke, han)
row_0               
(chewbacca, luke)   0.0 1.0 0.0 1.0
(chewbacca, obi)    0.5 0.0 0.0 0.0
(leia, luke)        0.5 0.0 0.0 0.0
(luke, han)         0.0 0.0 1.0 0.0

我还创建了一个函数

from itertools import islice

def my_function(seq, n = 2):
it = iter(seq)
result = tuple(islice(it, n))
if len(result) == n:
    yield result
for elem in it:
    result = result[1:] + (elem,)
    yield result

应用函数并计算概率

pairs = pd.DataFrame(my_function(star_wars), columns=['Columns', 'Rows'])
counts = pairs.groupby('Columns')['Rows'].value_counts()
probs = (counts/counts.sum()).unstack()

print(probs)

但是它给了我成对的计算(甚至不确定它是正确的)

But it gives me the calculation of pairs (not even sure it is correct)

Rows               (chewbacca, luke)  (chewbacca, obi)  (leia, luke)  \
Columns                                                                
(chewbacca, luke)                NaN               0.2           0.2   
(chewbacca, obi)                 0.2               NaN           NaN   
(darth, leia)                    NaN               NaN           NaN   
(luke, han)                      0.2               NaN           NaN   

Rows               (luke, han)  
Columns                         
(chewbacca, luke)          NaN  
(chewbacca, obi)           NaN  
(darth, leia)              0.2  
(luke, han)                NaN  

另一种尝试，仅使用 crosstab

所需-具有概率而不是数字的矩阵.

例如

            chewbacca  darth  han  leia  luke  obi
chewbacca          0      0    0     0   0.66 0.33
darth              0      0    0     1     0    0
han                0      0    0     0     1    0
leia               0      0    0     0     1    0
luke               0      0    0     0     0    0
obi                0      0    0     0     0    0

感谢您的时间和帮助！

推荐答案

我们仍然可以通过 crosstab

df=pd.DataFrame(star_wars)
s=pd.crosstab(df[0],df[1],normalize='index')
s=s.reindex(index=df.stack().unique(),fill_value=0).reindex(columns=df.stack().unique(),fill_value=0)
s
1          darth  leia      luke  han  chewbacca       obi
0                                                         
darth          0   1.0  0.000000  0.0          0  0.000000
leia           0   0.0  1.000000  0.0          0  0.000000
luke           0   0.0  0.000000  1.0          0  0.000000
han            0   0.0  0.000000  0.0          0  0.000000
chewbacca      0   0.0  0.666667  0.0          0  0.333333
obi            0   0.0  0.000000  0.0          0  0.000000

这篇关于仅使用numpy和pandas计算转换矩阵中每个单词的频率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！