Python，字典和卡方应急表 [英] Python, dictionaries, and chi-square contingency table

问题描述

这是一个我一直在大脑里呆了很久的问题，所以任何帮助都会很棒。我有一个文件，其中包含以下格式的几行（单词，发生单词的时间，以及在给定实例中包含给定单词的文档的时间）。以下是输入文件的样例。

This is a problem I've been racking my brains on for a long time, so any help would be great. I have a file which contains several lines in the following format (word, time that the word occurred in, and frequency of documents containing the given word within the given instance in time). Below is an example of what the inputfile looks like.

#inputfile
<word, time, frequency>
apple, 1, 3
banana, 1, 2
apple, 2, 1
banana, 2, 4
orange, 3, 1

我下面有Python类，我用来创建2-D字典来存储上面的文件作为关键字，而频率作为值：

I have Python class below that I used to create 2-D dictionaries to store the above file using as the key, and frequency as the value:

class Ddict(dict):
    '''
    2D dictionary class
    '''
    def __init__(self, default=None):
            self.default = default

    def __getitem__(self, key):
            if not self.has_key(key):
                self[key] = self.default()
            return dict.__getitem__(self, key)


wordtime=Ddict(dict) # Store each inputfile entry with a <word,time> key
timeword=Ddict(dict) # Store each inputfile entry with a <time,word> key

# Loop over every line of the inputfile
for line in open('inputfile'):
    word,time,count=line.split(',')

    # If <word,time> already a key, increment count
    try:
        wordtime[word][time]+=count
    # Otherwise, create the key
    except KeyError:
        wordtime[word][time]=count

    # If <time,word> already a key, increment count     
    try:
        timeword[time][word]+=count
    # Otherwise, create the key
    except KeyError:
        timeword[time][word]=count

我曾经在迭代条目时计算某些事情的问题在这个2D字典。对于每个时间't'的每个单词w，计算：

The question that I have pertains to calculating certain things while iterating over the entries in this 2D dictionary. For each word 'w' at each time 't', calculate:

1. 与 $在时间't'内的b $ b字'w'。 （a）


2. 在时间't'内的文件数不含
   字'w' （b）


3. 与
   文字'w'外部时间't'的文件数。 （c）


4. 没有
   文字'w'的文件数量时间't'。 （d）

1. The number of documents with word 'w' within time 't'. (a)
2. The number of documents without word 'w' within time 't'. (b)
3. The number of documents with word 'w' outside time 't'. (c)
4. The number of documents without word 'w' outside time 't'. (d)

上述每个项目代表每个单词和时间的卡方异常表的一个单元格。所有这些都可以在单个循环中计算，还是需要一次完成一次？

Each of the items above represents one of the cells of a chi-square contingency table for each word and time. Can all of these be calculated within a single loop or do they need to be done one at a time?

理想情况下，我希望输出是下面的内容，其中a，b，c，d都是上面计算的项目：

Ideally, I would like the output to be what's below, where a,b,c,d are all the items calculated above:

print "%s, %s, %s, %s" %(a,b,c,d)

在上述输入文件的情况下，在'1'时尝试找到apple一词的应急表的结果将是（3,2,1,6）。我将解释每个单元格的计算方式：

In the case of the input file above, the result of trying to find the contingency table for the word 'apple' at time '1' would be (3,2,1,6). I'll explain how each cell is calculated:

• '3'文件包含
  时间'1'内的'apple'。


• 在时间内有'2'文件
  '1'不包含'apple'。


• 有'1' '文件包含
  '苹果'外部时间'1'。


• 在时间之外有6个文件
  '1'不包含单词
  'apple'（1 + 4 + 1）。

• '3' documents contain 'apple' within time '1'.
• There are '2' documents within time '1' that don't contain 'apple'.
• There is '1' document containing 'apple' outside time '1'.
• There are 6 documents outside time '1' that don't contain the word 'apple' (1+4+1).

推荐答案

/ 1加起来12，超过观察总数（11）！在'1'之外只有5个文件不包含apple一词。

Your 4 numbers for apple/1 add up to 12, more than the total number of observations (11)! There are only 5 documents outside time '1' that don't contain the word 'apple'.

您需要将观察结果分为4个不相交的子集：

a：apple和1 => 3

b：not-apple和1 => 2

c：apple and not-1 => 1

d ：not-apple and not-1 => 5

You need to partition the observations into 4 disjoint subsets:
a: apple and 1 => 3
b: not-apple and 1 => 2
c: apple and not-1 => 1
d: not-apple and not-1 => 5

这是一些代码，显示一种方法：

Here is some code that shows one way of doing it:

from collections import defaultdict

class Crosstab(object):

    def __init__(self):
        self.count = defaultdict(lambda: defaultdict(int))
        self.row_tot = defaultdict(int)
        self.col_tot = defaultdict(int)
        self.grand_tot = 0

    def add(self, r, c, n):
        self.count[r][c] += n
        self.row_tot[r] += n
        self.col_tot[c] += n
        self.grand_tot += n

def load_data(line_iterator, conv_funcs):
    ct = Crosstab()
    for line in line_iterator:
        r, c, n = [func(s) for func, s in zip(conv_funcs, line.split(','))]
        ct.add(r, c, n)
    return ct

def display_all_2x2_tables(crosstab):
    for rx in crosstab.row_tot:
        for cx in crosstab.col_tot:
            a = crosstab.count[rx][cx]
            b = crosstab.col_tot[cx] - a
            c = crosstab.row_tot[rx] - a
            d = crosstab.grand_tot - a - b - c
            assert all(x >= 0 for x in (a, b, c, d))
            print ",".join(str(x) for x in (rx, cx, a, b, c, d))

if __name__ == "__main__":

    # inputfile
    # <word, time, frequency>
    lines = """\
    apple, 1, 3
    banana, 1, 2
    apple, 2, 1
    banana, 2, 4
    orange, 3, 1""".splitlines()

    ct = load_data(lines, (str.strip, int, int))
    display_all_2x2_tables(ct)

，这里是输出：

orange,1,0,5,1,5
orange,2,0,5,1,5
orange,3,1,0,0,10
apple,1,3,2,1,5
apple,2,1,4,3,3
apple,3,0,1,4,6
banana,1,2,3,4,2
banana,2,4,1,2,4
banana,3,0,1,6,4

这篇关于Python，字典和卡方应急表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！