对于Python/Linux，最大线程限制实际上是否无关紧要? [英] Is the max thread limit actually a non-relevant issue for Python / Linux?

问题描述

我正在使用的当前Python应用程序需要利用1000多个线程(Python线程模块).并不是说任何单个线程都在最大cpu周期内工作，这只是我正在创建的Web服务器负载测试应用程序. IE.模拟200个全部都渴望进入Web服务器并下载小型Web组件的firefox客户端，基本上模拟了几秒钟而不是几微秒的人员.

The current Python application that I'm working on has a need to utilize 1000+ threads (Pythons threading module). Not that any single thread is working at max cpu cycles, this is just a web server load test app I'm creating. I.E. emulate 200 firefox clients all longing into web server and downloading small web components, basically emulating humans that operate in seconds as opposed to microseconds.

因此，我正在阅读各种主题，例如"Linux/windows上python支持多少个线程等，我看到了很多不同的答案.一个用户说的全部都是关于内存和Linux内核的默认设置

So, I was reading through the various topics such as "how many threads does python support on Linux / windows, etc, and I saw a lot of varied answers. One users said its all about memory and the Linux kernel by default only sets aside 8Meg for threads, if it exceeds that then threads start being killed by the Kernel.

一个人说这对于CPython来说不​​是问题，因为无论如何一次仅运行一个线程(由于GIL)，所以我们可以指定一个庞大的线程???实际的真实情况是什么?

One guy stated this is a non issue for CPython because only 1 thread is running at a time anyway (because of the GIL) so we can specify a gazillion threads??? What's the actual truth on this?

推荐答案

1. 由于GIL，一次运行一个线程."好吧，有点. GIL意味着一次只能有一个线程可以执行 Python 代码.但是，任何数量的线程都可以执行IO，执行各种其他系统调用或其他不包含GIL的代码.

1. "One thread is running at a time because of the GIL." Well, sort of. The GIL means that only one thread can be executing Python code at a time. However, any number of threads could be doing IO, various other syscalls, or other code that doesn't hold the GIL.

听起来您的线程将主要执行网络I/O，并且任何数量的线程都可以同时执行I/O.拥有1000个线程的GIL竞争可能非常激烈，但是您始终可以创建多个Python进程并在它们之间划分I/O线程(即，在开始之前fork几次).

It sounds like your threads will be doing mostly network I/O, and any number of threads can do I/O simultaneously. The GIL competition might be pretty fierce with 1000 threads, but you can always create multiple Python processes and divide the I/O threads between them (i.e., fork a couple times before you start).

默认情况下，Linux内核仅为线程留出8Meg的空间."我不确定你在哪里听到的.也许您实际听到的是在Linux上，默认堆栈大小通常为8 MiB"，这是事实.每个线程将占用8 MiB的地址空间用于堆栈(在64位上没有问题)，以及用于额外的内存映射和线程进程本身的内核资源.您可以使用threading.stack_size库函数来更改堆栈大小，如果您有很多不进行深层调用的线程，则该功能会有所帮助.

"The Linux kernel by default only sets aside 8Meg for threads." I'm not sure where you heard that. Maybe what you actually heard was "On Linux, the default stack size is often 8 MiB," which is true. Each thread will use up 8 MiB of address space for stack (no problem on 64-bit) plus kernel resources for the additional memory maps and the thread process itself. You can change the stack size using the threading.stack_size library function, which helps if you have a lot of threads that don't make deep calls.

>>> import threading
>>> threading.stack_size()
0 # platform default, probably 8 MiB
>>> threading.stack_size(64*1024) # 64 KiB stack size for future threads

该线程中的其他建议使用异步/非阻塞框架.好吧，你可以做到.但是，在现代Linux内核上，多线程模型与异步(select/poll/epoll)I/O多路复用技术相比具有竞争优势.重写代码以使用异步模型是一项艰巨的工作，因此，只有在无法从线程模型获得所需性能的情况下，我才会这样做.如果您的线程确实在尝试模拟人为延迟(例如，将大部分时间都花在睡觉上)，那么在许多情况下，异步方法实际上要慢.我不确定这是否适用于Python，仅在GIL争用减少的情况下，是否值得进行切换.

Others in this thread have suggested using an asynchronous / nonblocking framework. Well, you can do that. However, on the modern Linux kernel, a multithreaded model is competitive with asynchronous (select/poll/epoll) I/O multiplexing techniques. Rewriting your code to use an asynchronous model is a non-trivial amount of work, so I'd only do it if I couldn't get the required performance from a threaded model. If your threads are really trying to simulate human latency (e.g., spend most of their time sleeping), there are a lot of scenarios in which the asynchronous approach is actually slower. I'm not sure if this applies to Python, where the reduced GIL contention alone might merit the switch.

这篇关于对于Python/Linux，最大线程限制实际上是否无关紧要?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！