多处理模块中的ThreadPool与Pool之间有什么区别? [英] What's the difference between ThreadPool vs Pool in the multiprocessing module?
问题描述
multiprocessing
模块中ThreadPool
和Pool
之间的区别是什么?当我尝试代码时,这是我看到的主要区别:
from multiprocessing import Pool
import os, time
print("hi outside of main()")
def hello(x):
print("inside hello()")
print("Proccess id: ", os.getpid())
time.sleep(3)
return x*x
if __name__ == "__main__":
p = Pool(5)
pool_output = p.map(hello, range(3))
print(pool_output)
我看到以下输出:
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
inside hello()
Proccess id: 13268
inside hello()
Proccess id: 11104
inside hello()
Proccess id: 13064
[0, 1, 4]
使用"ThreadPool":
from multiprocessing.pool import ThreadPool
import os, time
print("hi outside of main()")
def hello(x):
print("inside hello()")
print("Proccess id: ", os.getpid())
time.sleep(3)
return x*x
if __name__ == "__main__":
p = ThreadPool(5)
pool_output = p.map(hello, range(3))
print(pool_output)
我看到以下输出:
hi outside of main()
inside hello()
inside hello()
Proccess id: 15204
Proccess id: 15204
inside hello()
Proccess id: 15204
[0, 1, 4]
我的问题是:
-
为什么每次在
Pool
中都运行外部__main __()"? -
multiprocessing.pool.ThreadPool
不产生新进程吗?它只是创建新线程? -
如果是这样,使用
multiprocessing.pool.ThreadPool
与仅使用threading
模块有什么区别?
我在任何地方都没有找到ThreadPool
的官方文档,有人可以在哪里找到我吗?
multiprocessing.pool.ThreadPool
的行为与multiprocessing.Pool
相同,唯一的区别是使用线程而不是进程来运行工作程序逻辑.
看到的原因
hi outside of main()
使用multiprocessing.Pool
进行多次打印是由于该池将生成 5个独立进程.每个进程将初始化其自己的Python解释器并加载模块,从而导致顶层print
再次被执行.
请注意,仅当使用spawn
进程创建方法(仅Windows上可用的方法)时才会发生这种情况.如果您使用fork
one(Unix),则只会看到有关线程的消息打印一次.
multiprocessing.pool.ThreadPool
未记录,因为其实现从未完成.它缺少测试和文档.您可以在源代码中查看其实现.
我相信下一个自然的问题是:何时使用基于线程的池,何时使用基于进程的池?
经验法则是:
- IO绑定的作业->
multiprocessing.pool.ThreadPool
- CPU绑定的作业->
multiprocessing.Pool
- 混合作业->取决于工作量,由于流程隔离带来的优势,我通常更喜欢
multiprocessing.Pool
在Python 3上,您可能想看看 concurrent.future.Executor
池实现.
Whats the difference between ThreadPool
and Pool
in multiprocessing
module. When I try my code out, this is the main difference I see:
from multiprocessing import Pool
import os, time
print("hi outside of main()")
def hello(x):
print("inside hello()")
print("Proccess id: ", os.getpid())
time.sleep(3)
return x*x
if __name__ == "__main__":
p = Pool(5)
pool_output = p.map(hello, range(3))
print(pool_output)
I see the following output:
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
inside hello()
Proccess id: 13268
inside hello()
Proccess id: 11104
inside hello()
Proccess id: 13064
[0, 1, 4]
With "ThreadPool":
from multiprocessing.pool import ThreadPool
import os, time
print("hi outside of main()")
def hello(x):
print("inside hello()")
print("Proccess id: ", os.getpid())
time.sleep(3)
return x*x
if __name__ == "__main__":
p = ThreadPool(5)
pool_output = p.map(hello, range(3))
print(pool_output)
I see the following output:
hi outside of main()
inside hello()
inside hello()
Proccess id: 15204
Proccess id: 15204
inside hello()
Proccess id: 15204
[0, 1, 4]
My questions are:
why is the "outside __main__()" run each time in the
Pool
?multiprocessing.pool.ThreadPool
doesn't spawn new processes? It just creates new threads?If so whats the difference between using
multiprocessing.pool.ThreadPool
as opposed to justthreading
module?
I don't see any official documentation for ThreadPool
anywhere, can someone help me out where I can find it?
The multiprocessing.pool.ThreadPool
behaves the same as the multiprocessing.Pool
with the only difference that uses threads instead of processes to run the workers logic.
The reason you see
hi outside of main()
being printed multiple times with the multiprocessing.Pool
is due to the fact that the pool will spawn 5 independent processes. Each process will initialize its own Python interpreter and load the module resulting in the top level print
being executed again.
Note that this happens only if the spawn
process creation method is used (only method available on Windows). If you use the fork
one (Unix), you will see the message printed only once as for the threads.
The multiprocessing.pool.ThreadPool
is not documented as its implementation has never been completed. It lacks tests and documentation. You can see its implementation in the source code.
I believe the next natural question is: when to use a thread based pool and when to use a process based one?
The rule of thumb is:
- IO bound jobs ->
multiprocessing.pool.ThreadPool
- CPU bound jobs ->
multiprocessing.Pool
- Hybrid jobs -> depends on the workload, I usually prefer the
multiprocessing.Pool
due to the advantage process isolation brings
On Python 3 you might want to take a look at the concurrent.future.Executor
pool implementations.
这篇关于多处理模块中的ThreadPool与Pool之间有什么区别?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!