wso2esb中的等待线程问题 [英] WAITING threads issue in wso2esb
问题描述
您好，我正在研究wso2esb并将Active MQ用于消息队列。
使用大约3周后，ESB服务器挂起，并借助对ESB的JMX监视我发现它们中有大量处于等待状态的Java线程。
[EsbMonitoring] ******* ********** java线程属性*********************
[EsbMonitoring]线程数：8873
[EsbMonitoring ] DaemonThreadCount：104
[EsbMonitoring] PeakThreadCount：8992
[EsbMonitoring] TotalStartedThreadCount：16086123
最初，当我们启动ESB服务器时，它们大约有560个线程
[EsbMonitoring] ******** ********* java线程属性*********************
[EsbMonitoring]线程数：592
[EsbMonitoring] DaemonThreadCount：78
[EsbMonitoring] PeakThreadCount：592
[EsbMonitoring] TotalStartedThreadCount：1510
我使用
与ESB服务器的jstack进行了线程转储
ActiveMQ连接执行程序：tcp：//my-desktop/127.0.1.1：61616 @ 60595 prio = 10 tid = 0x00007fdb1c890800 nid = 0x259e等待状态[0x00007fda951cd000]
java.lang.Thread.State：等待（停车）
在sun.misc.Unsafe.park（本机方法）
-等待< 0x000000070bdf6c18>停车（java.util.concurrent.locks.AbstractQueuedSynchronizer $ ConditionObject）
在java.util.concurrent.locks.LockSupport.park（LockSupport.java:186）
在java.util.concurrent.locks。 AbstractQueuedSynchronizer $ ConditionObject.await（AbstractQueuedSynchronizer.java:2043）
在java.util.concurrent.LinkedBlockingQueue.take（LinkedBlockingQueue.java:442）
在java.util.concurrent.ThreadPoolExecutor.getTask（ThreadPoolExecutor。 java：1068）java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1130）
java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:615）$ b java.lang.Thread.run（Thread.java:745）上的
$ b ActiveMQ连接执行器：tcp：//my-desktop/127.0.1.1：61616 @ 60589 prio = 10 tid = 0x00007fdb1c8b7800 nid = 0x259a正在等待条件[0x00007fda950cc000]
java.lang.Thread.State：在sun.misc.Unsafe.park（本机方法）处等待（停车）
（本机方法）
-停车以等待< 0x000000070be68c48> （java.util.concurrent.locks.AbstractQueuedSynchronizer $ ConditionObject）
在java.util.concurrent.locks.LockSupport.park（LockSupport.java:186）
在java.util.concurrent.locks。 AbstractQueuedSynchronizer $ ConditionObject.await（AbstractQueuedSynchronizer.java:2043）
在java.util.concurrent.LinkedBlockingQueue.take（LinkedBlockingQueue.java:442）
在java.util.concurrent.ThreadPoolExecutor.getTask（ThreadPoolExecutor。 java：1068）java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1130）
java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:615）$ b java.lang.Thread.run（Thread.java:745）上的
$ b ActiveMQ连接执行器：tcp：//my-desktop/127.0.1.1：61616 @ 60590 prio = 10 tid = 0x00007fdb1c8ae800 nid = 0x2597在条件[0x00007fda946c2000]
java.lang.Thread.State上：等待（停车）
在sun.misc.Unsafe.park（本机方法）
-停车以等待< 0x000000070be22c88> （java.util.concurrent.locks.AbstractQueuedSynchronizer $ ConditionObject）
在java.util.concurrent.locks.LockSupport.park（LockSupport.java:186）
在java.util.concurrent.locks。 AbstractQueuedSynchronizer $ ConditionObject.await（AbstractQueuedSynchronizer.java:2043）
在java.util.concurrent.LinkedBlockingQueue.take（LinkedBlockingQueue.java:442）
在java.util.concurrent.ThreadPoolExecutor.getTask（ThreadPoolExecutor。 java：1068）java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1130）
java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:615）$ b java.lang.Thread.run（Thread.java:745）上的
$ b 所有这些连接都是在活动MQ端关闭了，但是ESB仍然保持这些线程并且使ESB麻烦，使消耗很高，达到死点后挂起。我需要重新启动ESB服务器以解决此问题。
首先，为什么这些连接永远处于WAITING状态，为什么ESB没有删除此类线程，有没有我可以在不重新启动ESB的情况下杀死或破坏等待线程的方法。
任何帮助将不胜感激。
谢谢您..！
另外几个问题
- 什么版本的WSO服务器？
- 什么是jvm版本？
无论如何，我认为您的负载很高。我认为我无法在短时间内轻松复制它。我有一些与postgresql jdbc驱动程序类似的东西，一段时间后连接被卡住，并且池中的所有连接都被阻塞。它仅在jdbc驱动程序更新中已修复。
根据
对于库依赖关系必须准确。也更新依赖关系。 jar文件列表在下面（来自官方文档 https：// docs。 wso2.com/display/ESB500/Configure+with+ActiveMQ ）
- activemq-broker-5.8.0 .jar
- activemq-client-5.8.0.jar
- activemq-kahadb -store-5.8.0.jar
- geronimo-jms_1.1_spec-1.1.1.jar
- geronimo-j2ee-management_1.1_spec-1.0.1.jar
- geronimo-jta_1.0.1B_spec-1.0.1.jar
- hawtbuf-1.9.jar
- slf4j-api-1.6.6.jar
- activeio-core-3.1.4.jar （可在/ lib / optional文件夹中找到）
另一种方法是更新到activemq代理的下一个版本。 ActiveMq是一个成熟的项目，因此在每个新版本中都不太可能引入大量的新功能并破坏与以前版本的兼容性。如果发行说明中没有任何异常之处，请尝试一下，在测试环境中进行测试并做出决定。
Hi am working on wso2esb and using Active MQ for message queues.
After around 3 weeks of usage ESB server was hanging and with the help of JMX monitoring of ESB i found that they are a huge number of java threads are in WAITING state.
[EsbMonitoring] ***************** java Threads Attributes *********************
[EsbMonitoring] ThreadCount :8873
[EsbMonitoring] DaemonThreadCount :104
[EsbMonitoring] PeakThreadCount :8992
[EsbMonitoring] TotalStartedThreadCount :16086123
Initially when we start the ESB server they are around 560 threads
[EsbMonitoring] ***************** java Threads Attributes *********************
[EsbMonitoring] ThreadCount :592
[EsbMonitoring] DaemonThreadCount :78
[EsbMonitoring] PeakThreadCount :592
[EsbMonitoring] TotalStartedThreadCount :1510
I took thread dump with jstack of ESB server with
"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616@60595" prio=10 tid=0x00007fdb1c890800 nid=0x259e waiting on condition [0x00007fda951cd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070bdf6c18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616@60589" prio=10 tid=0x00007fdb1c8b7800 nid=0x259a waiting on condition [0x00007fda950cc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070be68c48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Connection Executor: tcp://my-desktop/127.0.1.1:61616@60590" prio=10 tid=0x00007fdb1c8ae800 nid=0x2597 waiting on condition [0x00007fda946c2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070be22c88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
All this connections are closed in the active MQ side but ESB is still holding these threads and troubling ESB and making consumption high and after reaching curtain point it hangs . I need to restart the ESB server to solve this issue.
Firstly why are these connection are WAITING state forever and why ESB is not removing these kind of threads , Is there any way i can kill or destroy WAITING threads with out restarting ESB.
Any help would be appreciated.
Thank You..!
Another few questions
- What version of WSO server ?
- What is jvm version ?
Anyway, i assume you have pretty high load. I don't think i can reproduce it easy in short time. I had something similar with postgresql jdbc driver, connections were getting stuck after some time, and all connection in pool are got blocked. It was fixed just on jdbc driver update.
According http://mvnrepository.com/artifact/org.apache.activemq/activemq-broker version 5.8.0 quite old. It was released in 2013. There are some bug reports in activemq jira about connection leak. Few thing you can try.
Get latest version of library from jboss repository. Link you will find on same page as shown on picture underneath
Just be accurate with library dependencies. Update dependencies as well. List of jar files is underneath (It is from official documentation https://docs.wso2.com/display/ESB500/Configure+with+ActiveMQ)
- activemq-broker-5.8.0.jar
- activemq-client-5.8.0.jar
- activemq-kahadb-store-5.8.0.jar
- geronimo-jms_1.1_spec-1.1.1.jar
- geronimo-j2ee-management_1.1_spec-1.0.1.jar
- geronimo-jta_1.0.1B_spec-1.0.1.jar
- hawtbuf-1.9.jar
- slf4j-api-1.6.6.jar
- activeio-core-3.1.4.jar (available in /lib/optional folder)
Another way just update to next release of activemq broker. ActiveMq mature project, so it less possible that in every new release it introduces massive amount of new features and break compatibility with previous version. If release notes doesn't state anything extra ordinary, try it, test it in test environment and make decision.
这篇关于wso2esb中的等待线程问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
