多处理模块中的 ThreadPool 与 Pool 有什么区别? [英] What's the difference between ThreadPool vs Pool in the multiprocessing module?

问题描述

multiprocessing 模块中的ThreadPool 和Pool 有什么区别.当我尝试我的代码时，这是我看到的主要区别:

Whats the difference between ThreadPool and Pool in multiprocessing module. When I try my code out, this is the main difference I see:

from multiprocessing import Pool
import os, time

print("hi outside of main()")

def hello(x):
    print("inside hello()")
    print("Proccess id: ", os.getpid())
    time.sleep(3)
    return x*x

if __name__ == "__main__":
    p = Pool(5)
    pool_output = p.map(hello, range(3))

    print(pool_output)

我看到以下输出:

hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
inside hello()
Proccess id:  13268
inside hello()
Proccess id:  11104
inside hello()
Proccess id:  13064
[0, 1, 4]

使用线程池":

from multiprocessing.pool import ThreadPool
import os, time

print("hi outside of main()")

def hello(x):
    print("inside hello()")
    print("Proccess id: ", os.getpid())
    time.sleep(3)
    return x*x

if __name__ == "__main__":
    p = ThreadPool(5)
    pool_output = p.map(hello, range(3))

    print(pool_output)

我看到以下输出:

hi outside of main()
inside hello()
inside hello()
Proccess id:  15204
Proccess id:  15204
inside hello()
Proccess id:  15204
[0, 1, 4]

我的问题是:

• 为什么每次在Pool中都会运行outside __main__()"?

• why is the "outside __main__()" run each time in the Pool?

multiprocessing.pool.ThreadPool 不会产生新进程?它只是创建新线程?

multiprocessing.pool.ThreadPool doesn't spawn new processes? It just creates new threads?

如果是这样，使用 multiprocessing.pool.ThreadPool 与仅使用 threading 模块有什么区别?

If so whats the difference between using multiprocessing.pool.ThreadPool as opposed to just threading module?

我在任何地方都没有看到任何关于 ThreadPool 的官方文档，有人可以帮我看看在哪里可以找到它吗?

I don't see any official documentation for ThreadPool anywhere, can someone help me out where I can find it?

推荐答案

multiprocessing.pool.ThreadPool 的行为与 multiprocessing.Pool 相同，唯一的区别是使用线程而不是进程来运行工作者逻辑.

The multiprocessing.pool.ThreadPool behaves the same as the multiprocessing.Pool with the only difference that uses threads instead of processes to run the workers logic.

你看到的原因

hi outside of main()

使用 multiprocessing.Pool 多次打印是因为池将 spawn 5 个独立进程.每个进程都会初始化自己的 Python 解释器并加载模块，从而导致顶层 print 再次执行.

being printed multiple times with the multiprocessing.Pool is due to the fact that the pool will spawn 5 independent processes. Each process will initialize its own Python interpreter and load the module resulting in the top level print being executed again.

请注意，只有在使用 spawn 进程创建方法时才会发生这种情况(仅适用于 Windows 的方法).如果您使用 fork 之一(Unix)，您将看到消息只打印一次，就像线程一样.

Note that this happens only if the spawn process creation method is used (only method available on Windows). If you use the fork one (Unix), you will see the message printed only once as for the threads.

multiprocessing.pool.ThreadPool 没有记录，因为它的实现从未完成.它缺乏测试和文档.你可以在源代码中看到它的实现.

The multiprocessing.pool.ThreadPool is not documented as its implementation has never been completed. It lacks tests and documentation. You can see its implementation in the source code.

我相信下一个自然问题是:何时使用基于线程的池以及何时使用基于进程的池?

I believe the next natural question is: when to use a thread based pool and when to use a process based one?

经验法则是:

• IO 绑定作业 ->multiprocessing.pool.ThreadPool
• CPU 绑定作业 ->multiprocessing.Pool
• 混合工作 ->取决于工作量，由于进程隔离带来的优势，我通常更喜欢 multiprocessing.Pool

在 Python 3 上，您可能需要查看 concurrent.future.Executor 池实现.

On Python 3 you might want to take a look at the concurrent.future.Executor pool implementations.

这篇关于多处理模块中的 ThreadPool 与 Pool 有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！