Python，多线程速度太慢，多进程 [英] Python, multithreading too slow, multiprocess

问题描述

我是多处理新手，

我对线程有所了解，但是我需要提高这种计算的速度，希望可以使用多处理:

I know something about threading but I need to increase the speed of this calculation, hopefully with multiprocessing:

示例描述::将字符串发送到线程，更改字符串+基准测试， 将结果发送回打印.

Example Description: sends string to a thread, alters string + benchmark test, send result back for printing.

from threading import Thread

class Alter(Thread):
    def __init__(self, word):
        Thread.__init__(self)
        self.word = word
        self.word2 = ''

    def run(self):
        # Alter string + test processing speed
        for i in range(80000):
            self.word2 = self.word2 + self.word

# Send a string to be altered
thread1 = Alter('foo')
thread2 = Alter('bar')
thread1.start()
thread2.start()

#wait for both to finish
while thread1.is_alive() == True: pass
while thread2.is_alive() == True: pass


print(thread1.word2)
print(thread2.word2)

这目前大约需要6秒钟，我需要它加快速度.
我一直在研究多处理，找不到与上述代码等效的东西.我认为我追求的是 pooling ，但是我发现的示例很难理解.我想利用所有内核(8个内核)multiprocessing.cpu_count()的优势，但实际上我只是获得了有关多处理的有用信息的片段，不足以重复上述代码.如果有人能指出我正确的方向或更好的方向，请提供一个示例，将不胜感激.请使用Python 3

This is currently takes about 6 seconds and I need it to go faster.
I have been looking into multiprocessing and cannot find something equivalent to the above code. I think what I am after is pooling but examples I have found have been hard to understand. I would like to take advantage of all cores (8 cores) multiprocessing.cpu_count() but I really just have scraps of useful information on multiprocessing and not enough to duplicate the above code. If anyone can point me in the right direction or better yet, provide an example that would be greatly appreciated. Python 3 please

推荐答案

只需将threading替换为multiprocessing，将Thread替换为Process.由于存在严重的GIL，Pyton中的线程几乎(几乎)从未用于获得性能！我在另一个 SO-post 中进行了解释，其中包含指向文档的链接以及一个

Just replace threading with multiprocessing and Thread with Process. Threads in Pyton are (almost) never used to gain performance because of the big bad GIL! I explained it in an another SO-post with some links to documentation and a great talk about threading in python.

但是多重处理模块故意与线程化非常相似模块.您几乎可以将其用作嵌入式替代品！

But the multiprocessing module is intentionally very similar to the threading module. You can almost use it as an drop-in replacement!

多重处理模块不提供AFAIK提供的功能来强制使用特定数量的内核.它依赖于OS实现.您可以使用Pool对象，并将工作对象限制为核心数量.或者，您可以寻找其他MPI库，例如pypar.在Linux下，您可以使用外壳下的管道在不同的内核上启动多个实例

The multiprocessing module doesn't AFAIK offer a functionality to enforce the use of a specific amount of cores. It relies on the OS-implementation. You could use the Pool object and limit the worker-onjects to the core-count. Or you could look for an other MPI library like pypar. Under Linux you could use a pipe under the shell to start multiple instances on different cores

这篇关于Python，多线程速度太慢，多进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！