Zipf分布:如何测量Zipf分布 [英] Zipf Distribution: How do I measure Zipf Distribution

问题描述

如何测量或找到Zipf分布?例如，我有一个英语单词的语料库.如何找到Zipf分布?我需要找到Zipf分布，然后绘制它的图形.但是我被困在寻找Zipf分布的第一步.

How do I measure or find the Zipf distribution ? For example, I have a corpus of english words. How do I find the Zipf distribution ? I need to find the Zipf ditribution and then plot a graph of it. But I am stuck in the first step which is to find the Zipf distribution.

从每个单词的频率计数来看，很明显，它遵循Zipf律.但是我的目的是绘制一个zipf分布图.我不知道如何计算分布图的数据

From the frequency count of each word, it is clear that it obeys the Zipf law. But my aim is to plot a zipf distribution graph. I have no idea about how to calculate the data for the distribution graph

推荐答案

我不假装理解统计信息.但是，根据从犯罪网站 ，这是python中的幼稚尝试.

I don't pretend to understand statistics. However, based upon reading from scipy site, here is a naive attempt in python.

构建数据

首先，我们获取数据.例如，我们从国立医学图书馆MeSH(医学主题词)ASCII文件中下载数据

First we get our data. For example we download data from National Library of Medicine MeSH (Medical Subject Heading) ASCII file d2016.bin (28 MB).
Next, we open file, convert to string.

open_file = open('d2016.bin', 'r')
file_to_string = open_file.read()

接下来，我们在文件中找到各个单词并分离出单词.

Next we locate individual words in the file and separate out words.

words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)

最后，我们准备一个字典，将唯一的单词作为键，并将单词数作为值.

Finally we prepare a dict with unique words as key and word count as values.

for word in words:
    count = frequency.get(word,0)
    frequency[word] = count + 1

构建zipf分发数据
为了提高速度，我们将数据限制为1000个字.

Build zipf distribution data
For speed purpose we limit data to 1000 words.

n = 1000
frequency = {key:value for key,value in frequency.items()[0:n]}

此后，我们得到值的频率，转换为numpy数组，并使用numpy.random.zipf函数从zipf分布中提取样本.

After that we get frequency of values , convert to numpy array and use numpy.random.zipf function to draw samples from a zipf distribution.

分布参数a =2.作为示例，因为它必须大于1. 出于可见性考虑，我们将数据限制为50个采样点.

Distribution parameter a =2. as a sample as it needs to be greater than 1. For visibility purpose we limit data to 50 sample points.

s = frequency.values()
s = np.array(s)

count, bins, ignored = plt.hist(s[s<50], 50, normed=True)
x = np.arange(1., 50.)
y = x**(-a) / special.zetac(a)

最后绘制数据.

全部组合

import re
from operator import itemgetter
import matplotlib.pyplot as plt
from scipy import special
import numpy as np

#Get our corpus of medical words
frequency = {}
open_file = open('d2016.bin', 'r')
file_to_string = open_file.read()
words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)

#build dict of words based on frequency
for word in words:
    count = frequency.get(word,0)
    frequency[word] = count + 1

#limit words to 1000
n = 1000
frequency = {key:value for key,value in frequency.items()[0:n]}

#convert value of frequency to numpy array
s = frequency.values()
s = np.array(s)

#Calculate zipf and plot the data
a = 2. #  distribution parameter
count, bins, ignored = plt.hist(s[s<50], 50, normed=True)
x = np.arange(1., 50.)
y = x**(-a) / special.zetac(a)
plt.plot(x, y/max(y), linewidth=2, color='r')
plt.show()

图解

这篇关于Zipf分布:如何测量Zipf分布的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！