Python:无法启动新线程. < 100个活动线程 [英] Python: Can't start new thread. <100 active threads
问题描述
我遇到以下错误:
----- Match 93028: ------ Patch 5.11 ------78 Threads Active
----- Match 93029: ------ Patch 5.11 ------77 Threads Active
----- Match 93030: ------ Patch 5.11 ------76 Threads Active
----- Match 93031: ------ Patch 5.11 ------71 Threads Active
----- Match 93032: ------ Patch 5.11 ------55 Threads Active
----- Match 93033: ------ Patch 5.11 ------56 Threads Active
----- Match 93034: ------ Patch 5.11 ------57 Threads Active
----- Match 93035: ------ Patch 5.11 ------58 Threads Active
----- Match 93036: ------ Patch 5.11 ------59 Threads Active
Traceback (most recent call last):
File "pulldata.py", line 91, in <module>
getPatchData('5.11', '511')
File "pulldata.py", line 64, in getPatchData
matchThread.start()
File "/usr/lib/python3.4/threading.py", line 850, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
通常这是由于打开了太多线程导致的，但是如您所见，我也在打印活动线程的数量.有< 100个活动线程，因此我不确定是什么问题.这是相关代码:
Normally this is caused by having too many threads open but as you can see I'm also printing the number of threads active. There are <100 active threads so I'm not sure what the problem is. Here is the relevant code:
slot = threading.BoundedSemaphore(value=1000)
def getMatchData(index,match,patch):
global requestsSent
global logfile
print("----- Match {0}: ------ Patch {1} ------{2} Threads Active".format(index,patch,threading.active_count()))
logfile.write("Parsing Match {0} for patch {1}:\n".format(index,patch))
#match is a class. get is a function that sends a request to the server and returns a request object from where I get the json response.
data = match.get().json()
#processdata
slot.release()
def getPatchData(patch, name):
global logfile
threads = []
matches = getAllMatches(patch)
for index, match in enumerate(matches):
slot.acquire()
matchThread = threading.Thread(target=getMatchData, args=(index,match,patch))
threads.append(matchThread)
matchThread.start()
for t in threads:
if not t.isAlive():
threads.remove(t)
for t in threads:
t.join()
插槽信号量应该限制活动线程的数量，但是我认为我从来没有达到1000个线程.在我假定此错误是由于线程数组所指向的线程引起的，所以我添加了代码，以便在它们不再处于活动状态时将其从数组中删除.
The slots semaphore is supposed to limit the number of active threads but I don't think I ever reached 1000 threads anyway. Before I assumed this error was being caused due to threads being pointed to by my array of threads so I added code to remove them from the array when they were no longer active.
我不明白为什么只有59个活动线程时为什么我无法启动新线程.
I can't understand why I get can't start a new thread when there are only 59 active threads.
还有，有没有更好的方法可以实现我想要做的事情?每个线程向API发送一个请求.我尝试在没有并发的情况下执行此操作，但我什至没有达到我的速率限制.
Also, is there a better way to achieve what I am trying to do? Each thread sends a request to an API. I tried doing it without concurrency but I wasn't even coming close to my rate limit.
推荐答案
我遇到了类似的问题，这就是我的解决方法.
I ran into a similar issue and here is how I solved it.
不知道OP使用的是什么操作系统，但是在Linux上通常每个用户的进程数受到限制.
您可以使用ulimit -u(或ulimit -a)查看它.
这个定义有点用词不当，因为限制实际上是在 OS线程数(或LWP)上.
(请参见以下接受的答案: https://superuser.com/questions/376532/does-gnu-linux-counts-processes-and-threads-together-when-i-limit-their-number )
Not sure what OS the OP is using, but on Linux there is usually a limit on the number of processes per user.
You can see it with ulimit -u (or also ulimit -a).
The definition is a bit of a misnomer, as the limit is actually on the number of OS threads (or LWP).
(see the accepted answer at: https://superuser.com/questions/376532/does-gnu-linux-counts-processes-and-threads-together-when-i-limit-their-number)
在我的系统上，限制似乎设置为400(但可以由管理员更改).
On my system, the limit appears to be set to 400 (but it can be changed by the admin).
您可以使用以下命令查看所有线程的列表:
You can see a list of all your threads with the command:
ps -fLu <your_username>
在我的情况下，我的python应用程序将引发与OP报告的异常相同的事件，但是threading.active_count()将返回7.
In my case, my python application would raise the same exception as reported by the OP, but threading.active_count() would return 7.
原来，我在以前的会话中有很多遗留的进程(我对nohup ...有点太热衷了)，每个线程都挂在系统中.删除它们消除了线程创建错误.
It turned out I had lots of left-over processes from previous sessions (I had been a bit too keen with nohup...), with several threads each, hanging around in the system. Removing them got rid of the thread creation error.
这篇关于Python:无法启动新线程. < 100个活动线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
