如何限制Python线程? [英] How can I throttle Python threads?

问题描述

我有一个线程正在执行大量CPU密集型处理，这似乎阻塞了其他线程.如何限制它?

I have a thread doing a lot of CPU-intensive processing, which seems to be blocking out other threads. How do I limit it?

这是专门针对web2py的，但是一般的解决方案就可以了.

This is for web2py specifically, but a general solution would be fine.

推荐答案

不久前，我实际上刚刚陷入了这个问题，您将无法更改线程优先级，但是有很多方法可以解决此问题.

I actually just ended up diving into this issue not long ago, you wont be able to change the thread priority but there are ways around this.

为了给您一些有关该问题的背景知识，在cPython实现中，由于释放和获取全局解释器锁定或GIL的方式，CPU绑定的线程可能会导致其他线程饿死.奇怪的是，在多核环境中，此问题变得更加严重. David Beazley 对此问题进行了非常详细的分析和介绍，您可以在

To give you a bit of background on the problem, in the cPython implementation CPU bound threads can cause other threads to starve because of the way the Global Interpreter Lock or GIL is released and acquired. Oddly enough this problem is made worse in a multicore environment. A really detailed analysis and presentation on this issue was done by David Beazley which you can find at http://www.dabeaz.com/python/GIL.pdf. He has several blog posts that go into more detail. They're long but quite fascinating.

简短的版本是，CPU绑定线程可以释放并重新获取GIL，然后才能唤醒其他线程来抢夺它.结果导致CPU绑定线程持有GIL的时间超过90％.

The short version is that the CPU bound thread releases and reacquires the GIL before the other threads can be woken up to grab it. Resulting in the CPU bound thread holding the GIL for more than 90% of the time.

可以使用一些模式来解决此问题.例如，您可以在完全不同的过程中运行与CPU绑定的任务.这将使操作系统调度程序可以更好地管理资源共享，并且应该允许您的web2py线程继续运行，因为操作系统实际上对IO绑定线程给予了优先待遇.针对此类情况，提供了 multiprocessing 库.它将需要更多代码才能使其正常运行，但这应该会有所帮助.

There are some patterns you can use to work around this issue. For example you can run your CPU bound tasks in a completely different process. This will allow the operating system scheduler to manage resource sharing a lot better and should allow your web2py threads to continue to run since operating systems actually give preferential treatment to IO bound threads. The multiprocessing library is provided for cases such as this. It will require some more code to get it working but that should help.

这篇关于如何限制Python线程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！