python线程和性能? [英] python threading and performance?

问题描述

我必须执行繁重的I/O绑定操作，即解析大型文件并将其从一种格式转换为另一种格式.最初，我以前是串行执行的，即一个接一个地解析..！性能非常差(用时超过90秒).因此，我决定使用线程来提高性能.我为每个文件创建了一个线程. (4个线程)

I had to do heavy I/o bound operation, i.e Parsing large files and converting from one format to other format. Initially I used to do it serially, i.e parsing one after another..! Performance was very poor ( it used take 90+ seconds). So I decided to use threading to improve the performance. I created one thread for each file. ( 4 threads)

 for file in file_list:
            t=threading.Thread(target = self.convertfile,args = file)
            t.start()
            ts.append(t)
 for t in ts:
            t.join()

但是令我惊讶的是，性能没有任何改善.现在也需要大约90秒钟以上的时间才能完成任务.由于这是I/O绑定操作，因此我希望可以提高性能.

But for my astonishment, there is no performance improvement whatsoever. Now also it takes around 90+ seconds to complete the task. As this is I/o bound operation , I had expected to improve the performance.

推荐答案

在普通的Python解释器下，由于

Under the usual Python interpreter, threading will not allocate more CPU cores to your program because of the global interpreter lock (aka. the GIL).

多重处理模块可以在这里为您提供帮助. (请注意，它是在Python 2.6中引入的，但对于Python 2.5存在反向移植.)

The multiprocessing module could help you out here. (Note that it was introduced in Python 2.6, but backports exist for Python 2.5.)

正如MSalters所说，如果您的程序受I/O约束，是否有用则值得商bat.但这可能值得一试:)

As MSalters says, if your program is I/O bound it's debatable whether this is useful. But it might be worth a shot :)

要使用此模块实现所需的目标，

To achieve what you want using this module:

import multiprocessing

MAX_PARALLEL_TASKS = 8 # I have an Intel Core i7 :)

pool = multiprocessing.Pool(MAX_PARALLEL_TASKS)

pool.map_async(convertfile, filelist)

pool.close()
pool.join()

重要！.传递给map_async的函数必须是可腌制的.通常，除非您将实例方法设计为可腌制的，否则实例方法是不可腌制的！请注意，上面的convertfile是一个函数.

Important! The function that you pass to map_async must be pickleable. In general, instance methods are NOT pickleable unless you engineering them to be so! Note that convertfile above is a function.

如果您实际上需要从convertfile处获取结果，也可以使用一些方法.多处理文档页面上的示例应阐明.

If you actually need to get results back from convertfile, there are ways to do that as well. The examples on the multiprocessing documentation page should clarify.

这篇关于python线程和性能?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！