ProcessPoolExecutor 和 ThreadPoolExecutor 有什么区别? [英] What is the difference between ProcessPoolExecutor and ThreadPoolExecutor?

问题描述

我阅读了试图获得基本理解的文档，但它只表明 ProcessPoolExecutor 允许回避 Global Interpreter Lock 我认为这是锁定的方法一个变量或函数，以便并行进程不会同时更新其值.

I read the docs trying to get a basic understanding but it only shows that ProcessPoolExecutor allows to side-step the Global Interpreter Lock which I think is the way to lock a variable or function so that parallel processes do not update its value at the same time.

我正在寻找的是何时使用 ProcessPoolExecutor 以及何时使用 ThreadPoolExecutor 以及在使用每种方法时我应该记住什么！

What I am looking for is when to use ProcessPoolExecutor and when to use ThreadPoolExecutor and what should I keep in mind while using each approach!

推荐答案

ProcessPoolExecutor 在自己独立的子进程中运行每个工作线程.

ProcessPoolExecutor runs each of your workers in its own separate child process.

ThreadPoolExecutor 在主进程内的不同线程中运行每个工作线程.

ThreadPoolExecutor runs each of your workers in separate threads within the main process.

全局解释器锁 (GIL) 不只是锁定变量或函数；它锁定了整个解释器.这意味着每个内置操作，包括诸如 listodicts[3]['spam'] = egg 之类的操作，都是自动线程安全的.

The Global Interpreter Lock (GIL) doesn't just lock a variable or function; it locks the entire interpreter. This means that every builtin operation, including things like listodicts[3]['spam'] = eggs, is automatically thread-safe.

但这也意味着，如果您的代码受 CPU 限制(即，它花时间进行计算而不是等待网络响应)，而不是将大部分时间花在旨在发布的外部库中GIL(如 NumPy)，一次只有一个线程可以拥有 GIL.所以，如果你有 4 个线程，即使你有 4 个甚至 16 个内核，大多数时候，其中 3 个将坐在那里等待 GIL.因此，您的代码速度不是提高了 4 倍，而是变慢了一点.

But it also means that if your code is CPU-bound (that is, it spends its time doing calculations rather than, e.g., waiting on network responses), and not spending most of its time in an external library designed to release the GIL (like NumPy), only one thread can own the GIL at a time. So, if you've got 4 threads, even if you have 4 or even 16 cores, most of the time, 3 of them will be sitting around waiting for the GIL. So, instead of getting 4x faster, your code gets a bit slower.

同样，对于 I/O 绑定的代码(例如，等待一堆服务器响应你发出的一堆 HTTP 请求)，线程就好了；仅对于受 CPU 限制的代码而言，这是一个问题.

Again, for I/O-bound code (e.g., waiting on a bunch of servers to respond to a bunch of HTTP requests you made), threads are just fine; it's only for CPU-bound code that this is an issue.

每个单独的子进程都有自己单独的 GIL，所以这个问题就迎刃而解了——即使你的代码受 CPU 限制，使用 4 个子进程仍然可以让它的运行速度提高近 4 倍.

Each separate child process has its own separate GIL, so this problem goes away—even if your code is CPU-bound, using 4 child processes can still make it run almost 4x as fast.

但是子进程不共享任何变量.通常，这是一件好事——您将值(的副本)作为参数传递给您的函数，并返回(副本)值，进程隔离可确保您安全地执行此操作.但是偶尔(通常是出于性能原因，也有时是因为您传递的对象无法通过 pickle 复制)，这是不可接受的，因此您要么需要使用线程，要么使用multiprocessing 模块中更复杂的显式共享数据包装器.

But child processes don't share any variables. Normally, this is a good thing—you pass (copies of) values in as the arguments to your function, and return (copies of) values back, and the process isolation guarantees that you're doing this safely. But occasionally (usually for performance reasons, but also sometimes because you're passing around objects that can't be copied via pickle), this is not acceptable, so you either need to use threads, or use the more complicated explicit shared data wrappers in the multiprocessing module.

这篇关于ProcessPoolExecutor 和 ThreadPoolExecutor 有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！