更好的解决方案为Python Threading.Event半忙等待 [英] Better solution for Python Threading.Event semi-busy waiting

问题描述

我使用漂亮的标准Threading.Event：
主线程到达一个循环运行的点：

  event.wait（60）
  

请求之前的其他块直到回复为止然后启动：

  event.set（）
  

我希望主线程选择40秒，但情况并非如此。
从Python 2.7源lib / threading.py：

 ＃平衡行为：我们买不起纯忙的循环，所以我们
＃必须睡觉;但是如果我们睡了整个超时时间，
＃我们将无法响应。这里的计划开始时非常少
＃，而且随着时间的推移，时间越来越长，但是每秒不超过20次（或者剩下的超时时间）。 
 endtime = _time（）+ timeout 
 delay = 0.0005＃500 us  - >初始延迟1 ms 
，而True：
 getit = waiter.acquire（0）
如果有：
 break 
 remaining = endtime  -  _time（）
如果剩余< = 0：
 break 
 delay = min（延迟* 2，剩余，.05）
 _sleep（延迟）
  / pre> 
 
 
我们得到的是一个select系统调用每500us运行一次。 
这会引起机器上显着的负载，而且选择循环非常紧张。
 
 
 
有人可以解释为什么会出现平衡行为，为什么它与一个线程等待文件描述符。
 
 
 
第二，有没有更好的方法来实现一个主要睡眠的主线程，没有这么紧的循环？
解决方案
我最近受到同样的问题的打击，我也跟踪到线程中的这个确切的代码块/ code> module。
 
 
 
它吸吮。
 
 
 
解决方案是重载线程模块或迁移到 python3 ，这部分实现已经修复。
 
 
 
在我的情况下，迁移蟒蛇3本来是一个巨大的努力，所以我选择了前者。我所做的是：
 
 
我创建了一个快速的 .so  code> cython ），其界面为 pthread 。它包括python函数，它调用相应的 pthread_mutex _ * 函数，并链接到 libpthread 。具体来说，与我们感兴趣的任务最相关的功能是 pthread_mutex_timedlock 。
 
 
我创建了一个新的 threading2 模块，（并替换所有导入线程我的代码库中的 import threading2 ）。在 threading2 中，我从线程（锁定，条件，事件）以及队列我使用了很多（队列和 PriorityQueue ）。 锁定类是使用 pthread_mutex _ * 函数完全重新实现的，但其余的更容易 - 我简单地子类化原来的（例如 threading.Event ），并覆盖 __ init __ 来创建我的新的 / code>类型。其他工作正常。
 
 
新的 Lock 类型的实现非常类似于线程中的原始实现，但是我基于获取的新实现，我在 python3 的线程模块（其自然比上述平衡动作块简单得多）。这个部分很容易。
 
 
 
（Btw，我的案例中的结果是我的大规模多线程流程的30％加速，甚至超过我预期的）。 > 
I'm using pretty standard Threading.Event:
Main thread gets to a point where its in a loop that runs:
event.wait(60)
The other blocks on a request until a reply is available and then initiates a:
event.set()
I would expect the main thread to select for 40 seconds, but this is not the case.
From the Python 2.7 source Lib/threading.py:
# Balancing act:  We can't afford a pure busy loop, so we
# have to sleep; but if we sleep the whole timeout time,
# we'll be unresponsive.  The scheme here sleeps very
# little at first, longer as time goes on, but never longer
# than 20 times per second (or the timeout time remaining).
endtime = _time() + timeout
delay = 0.0005 # 500 us -> initial delay of 1 ms
while True:
   gotit = waiter.acquire(0)
   if gotit:
       break
   remaining = endtime - _time()
   if remaining <= 0:
       break
   delay = min(delay * 2, remaining, .05)
   _sleep(delay)
What we get is a select syscall run every 500us. 
This causes noticeable load on the machine with a pretty tight select loop.

Can someone please explain why there is a balancing act involved and why is it different than a thread waiting on a file descriptor.

and second, Is there a better way to implement a mostly sleeping main thread without such a tight loop?
解决方案
I recently got hit by the same problem, and I also tracked it down to this exact block of code in the threading module.

It sucks.

The solution would be to either overload the threading module, or migrate to python3, where this part of the implementation has been fixed.

In my case, migrating to python3 would have been a huge effort, so I chose the former.  What I did was:

I created a quick .so file (using cython) with an interface to pthread.  It includes python functions which invoke the corresponding pthread_mutex_* functions, and links against libpthread.  Specifically, the function most relevant to the task we're interested in is pthread_mutex_timedlock.
I created a new threading2 module, (and replaced all import threading lines in my codebase with import threading2).  In threading2, I re-defined all the relevant classes from threading (Lock, Condition, Event), and also ones from Queue which I use a lot (Queue and PriorityQueue).  The Lock class was completely re-implemented using pthread_mutex_* functions, but the rest were much easier -- I simply subclassed the original (e.g. threading.Event), and overridden __init__ to create my new Lock type. The rest just worked.
The implementation of the new Lock type was very similar to the original implementation in threading, but I based the new implemenation of acquire on the code I found in python3's threading module (which, naturally, is much simpler than the abovementioned "balancing act" block). This part was fairly easy.

(Btw, the result in my case was 30% speedup of my massively-multithreaded process.  Even more than I expected.)

                        
这篇关于更好的解决方案为Python Threading.Event半忙等待的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！