Python多重处理:读取大文件并更新导入的字典 [英] Python multiprocessing: Reading a large file and updating an imported dictionary

问题描述

我需要使用 multiprocessing Pool 和 Manager 来读取大文件并相应地更新导入的字典.这是我的代码:

I need to read a large file and update an imported dictionary accordingly, using multiprocessing Pool and Manager. Here is my code:

from multiprocessing import Pool, Manager

manager = Manager()
d = manager.dict()
imported_dic = json.load(~/file.json) #loading a file containing a large dictionary
d.update(imported_dic)

def f(line):
    data = line.split('\t')
    uid = data[0]
    tweet = data[2].decode('utf-8')

    if #sth in tweet:
        d[uid] += 1

p = Pool(4)
with open('~/test_1k.txt') as source_file:
    p.map(f, source_file)

但是它不能正常工作.知道我在这里做错什么吗?

But it does not work properly. Any idea what am I doing wrong here?

推荐答案

尝试以下代码:

d = init_dictionary( ) # some your magic here

def f(line):
    data = line.split('\t')
    uid = data[0]
    tweet = data[2].decode('utf-8')

    if uid in d: 
        for n in d[uid].keys(): 
            if n in tweet: 
                 yield uid, n, 1 
            else: 
                 yield uid, n, 0 


p = Pool(4)

with open('~/test_1k.txt') as source_file:
    for stat in p.map(f, source_file):
          uid, n, r = stat
          d[uid][n] += r

这是相同的解决方案，但没有共享字典.

It's same solution, but without shared dictionary.

这篇关于Python多重处理:读取大文件并更新导入的字典的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！