如何以多线程方式将Python Dict转换为JSON [英] How do I Convert Python Dict to JSON in a Multi-Threaded Fashion

问题描述

我有许多大型文件，其中有成千上万行采用python dict格式的文件.我正在将它们与json.dumps转换为json字符串.

I have a number of large files with many thousands of lines in python dict format. I'm converting them with json.dumps to json strings.

import json
import ast

mydict = open('input', 'r')
output = open('output.json', "a")

for line in mydict:
        line = ast.literal_eval(line)
        line = json.dumps(line)
        output.write(line)
        output.write("\n")

这可以完美地工作，但是它以单线程方式工作.有没有一种简单的方法可以利用系统中的其余内核来加快速度?

This works flawlessly, however, it does so in a single threaded fashion. Is there an easy way to utilize the remaining cores in my system to speed things up?

基于我在这里从多处理库开始的建议:

Based on the suggestions I've started here with the multiprocessing library:

import os
import json
import ast
from multiprocessing import Process, Pool

mydict = open('twosec.in', 'r')

def info(title):
        print title
        print 'module name:', __name__
        print 'parent process: ', os.getppid()
        print 'process id:', os.getpid()

def converter(name):
        info('converter function')
        output = open('twosec.out', "a")
        for line in mydict:
                line = ast.literal_eval(line)
                line = json.dumps(line)
                output.write(line)
                output.write("\n")

if __name__ == '__main__':
        info('main line')
        p = Process(target=converter, args=(mydict))
        p.start()
        p.join()

我不太了解Pool在哪里发挥作用，您能解释更多吗?

I don't quite understand where Pool comes into play, can you explain more?

推荐答案

将上面的代码包装到一个函数中，该函数将文件名作为单个参数，并将json写入输出文件.

Wrap the code above in a function that takes as its single argument a filename and that writes the json to an output file.

然后从multiprocessing模块创建一个Pool对象，并使用Pool.map()将您的函数并行应用于所有文件的列表.这将自动使用您CPU上的所有内核，并且由于它使用多个进程而不是线程，因此您不会遇到全局解释器锁.

Then create a Pool object from the multiprocessing module, and use Pool.map() to apply your function in parallel to the list of all files. This will automagically use all cores on your CPU, and because it uses multiple processes instead of threads, you won't run into the global interpreter lock.

编辑:像这样更改程序的主要部分；

Change the main portion of your program like so;

  if __name__ == '__main__':
     files = ['first.in', 'second.in', 'third.in'] # et cetera
     info('main line')
     p = Pool()
     p.map(convertor, files)
     p.close()

当然，您还应该更改convertor()以从输入名称中获取输出名称！

Of course you should also change convertor() to derive the output name from the input name!

以下是使用 ImageMagick 程序将DICOM文件转换为PNG格式的程序的完整示例. /p>

Below is a complete example of a program to convert DICOM files into PNG format, using the ImageMagick program

"Convert DICOM files to PNG format, remove blank areas."

import os
import sys # voor argv.
import subprocess
from multiprocessing import Pool, Lock

def checkfor(args):
    try:
        subprocess.check_output(args, stderr=subprocess.STDOUT)
    except CalledProcessError:
        print "Required program '{}' not found! exiting.".format(progname)
        sys.exit(1)

def processfile(fname):
    size = '1574x2048'
    args = ['convert', fname, '-units', 'PixelsPerInch', '-density', '300', 
            '-crop', size+'+232+0', '-page', size+'+0+0', fname+'.png']
    rv = subprocess.call(args)
    globallock.acquire()
    if rv != 0:
        print "Error '{}' when processing file '{}'.".format(rv, fname)
    else:
        print "File '{}' processed.".format(fname)
    globallock.release()

## This is the main program ##
if __name__ == '__main__':
    if len(sys.argv) == 1:
        path, binary = os.path.split(sys.argv[0])
        print "Usage: {} [file ...]".format(binary)
        sys.exit(0)
    checkfor('convert')
    globallock = Lock()
    p = Pool()
    p.map(processfile, sys.argv[1:])
    p.close()

这篇关于如何以多线程方式将Python Dict转换为JSON的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！