tomcat中有100个线程TIMED_WAITING，导致它在线程总数超过200时停止 [英] 100 threads TIMED_WAITING in tomcat, causing it to stall as total number of threads crosses 200

问题描述

最近我们的一个生产tomcat服务器没有响应，因为tomcat的繁忙线程高达200.当我们在重新启动之前进行线程转储时，我们在TIMED_WAITING状态下获得了100个线程，就像这3个线程一样：

Recently one of our production tomcat server became unresponsive because tomcat's busy threads shot upto 200. When we took thread dump before restarting we got 100 threads in TIMED_WAITING state like these 3 threads:

""http-bio-7007"-exec-241" daemon prio=10 tid=0x00002aaab107b000 nid=0x59df waiting on condition [0x0000000051239000]
java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000580d877d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

""http-bio-7007"-exec-237" daemon prio=10 tid=0x00002aaab186e000 nid=0x596d waiting on condition [0x000000004d1f9000]
java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000580d877d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

""http-bio-7007"-exec-236" daemon prio=10 tid=0x00002aaab1118000 nid=0x596c waiting on condition [0x000000004e50c000]
java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000580d877d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
   at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

我们有4个应用程序的线程池（例如pool-4-thread-20等），它们每个都有20个线程我不确定这100个线程在等待哪个阻塞队列？我们正在使用c3P0连接池和hibernate，这似乎不是原因。

We have 4 application's thread pools(e.g. pool-4-thread-20 etc) also which are having 20 threads each so I am not sure on which blocking queue these 100 threads waiting? We are using c3P0 connection pool with hibernate which doesn't seems to be cause of this.

知道java.util.concurrent.locks.AbstractQueuedSynchronizer $ ConditionObject是什么？

Any idea what java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject is?

推荐答案

当我们修复了泄漏由c3p0管理的数据库连接的代码时，这个问题得到了解决。我们的代码中很少有流，我们在关闭finally块中的实体管理器之前没有专门在catch块中调用rollback（），所以在异常情况下连接没有返回池并且异常频率很高（超过大小）在超时间隔内的池，然后所有其他进程线程将堆积起来以获得连接。

This got fixed when we fixed our code which was leaking DB connections managed by c3p0. There were few flows in our code where we were not calling rollback() specifically in catch block before closing entity Manager in finally block, so in case of exceptions connection was not coming back to pool and if frequency of exception is high(more than size of pool within timeout interval) then all other process threads would pile up to get connection.

这篇关于tomcat中有100个线程TIMED_WAITING，导致它在线程总数超过200时停止的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！