Python 线程都在单个内核上执行 [英] Python threads all executing on a single core

问题描述

我有一个 Python 程序，它产生许多线程，一次运行 4 个，每个执行一个昂贵的操作.伪代码:

I have a Python program that spawns many threads, runs 4 at a time, and each performs an expensive operation. Pseudocode:

for object in list:
    t = Thread(target=process, args=(object))
    # if fewer than 4 threads are currently running, t.start(). Otherwise, add t to queue

但是当程序运行时，OS X 中的 Activity Monitor 显示 4 个逻辑核心中有 1 个为 100%，其他的接近 0.显然我不能强迫操作系统做任何事情，但我已经以前从未像这样关注多线程代码的性能，所以我想知道我是否只是遗漏或误解了某些东西.

But when the program is run, Activity Monitor in OS X shows that 1 of the 4 logical cores is at 100% and the others are at nearly 0. Obviously I can't force the OS to do anything but I've never had to pay attention to performance in multi-threaded code like this before so I was wondering if I'm just missing or misunderstanding something.

谢谢.

推荐答案

请注意，在许多情况下(以及几乎所有昂贵的操作"都是在 Python 中实现的计算的情况)，由于以下原因，多个线程实际上不会并发运行Python 的 全局解释器锁 (GIL).

Note that in many cases (and virtually all cases where your "expensive operation" is a calculation implemented in Python), multiple threads will not actually run concurrently due to Python's Global Interpreter Lock (GIL).

GIL 是解释器级别的锁.此锁可防止执行Python中的多个线程一次口译员.每个想要的线程运行必须等待 GIL由另一个线程释放，其中意味着你的多线程 Python应用程序基本上是单一的螺纹，对吗?是的.不完全是.有点.

The GIL is an interpreter-level lock. This lock prevents execution of multiple threads at once in the Python interpreter. Each thread that wants to run must wait for the GIL to be released by the other thread, which means your multi-threaded Python application is essentially single threaded, right? Yes. Not exactly. Sort of.

CPython 使用所谓的操作系统"线程在封面下，也就是说每次请求创建一个新线程，解释器实际上调用了操作系统的库和内核生成新线程.这个例如，与 Java 相同.所以在记忆中你确实有多个线程和通常的操作系统控制哪个线程是预定运行.在多个处理器机器，这意味着你可以有很多线程分布多个处理器，一切都很愉快忙着做工作.

CPython uses what’s called "operating system" threads under the covers, which is to say each time a request to make a new thread is made, the interpreter actually calls into the operating system’s libraries and kernel to generate a new thread. This is the same as Java, for example. So in memory you really do have multiple threads and normally the operating system controls which thread is scheduled to run. On a multiple processor machine, this means you could have many threads spread across multiple processors, all happily chugging away doing work.

然而，虽然 CPython 确实使用操作系统线程(理论上允许多个线程执行在口译员内同时)，口译员也强制 GIL 被一个线程之前，它可以访问解释器和堆栈，可以修改内存中的所有 Python 对象随意.后一点是为什么GIL 存在:GIL 阻止同时访问 Python 对象通过多个线程.但这不拯救你(如银行所示示例)从锁定敏感生物；你不能搭便车.GIL 是为了保护口译员的记忆，而不是你的理智.

However, while CPython does use operating system threads (in theory allowing multiple threads to execute within the interpreter simultaneously), the interpreter also forces the GIL to be acquired by a thread before it can access the interpreter and stack and can modify Python objects in memory all willy-nilly. The latter point is why the GIL exists: The GIL prevents simultaneous access to Python objects by multiple threads. But this does not save you (as illustrated by the Bank example) from being a lock-sensitive creature; you don’t get a free ride. The GIL is there to protect the interpreters memory, not your sanity.

请参阅Jesse Noller 的帖子 了解更多详情.

See the Global Interpreter Lock section of Jesse Noller's post for more details.

要解决这个问题，请查看 Python 的多处理模块.

To get around this problem, check out Python's multiprocessing module.

多个进程(明智地使用IPC)是[...]一个更好的为多 CPU 编写应用程序的方法盒子比线程.

multiple processes (with judicious use of IPC) are[...] a much better approach to writing apps for multi-CPU boxes than threads.

-- Guido van Rossum(Python的创造者)

这篇关于Python 线程都在单个内核上执行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！