在Spring Azure qPID JMS代码中跟踪内存泄漏 [英] Tracing memory leak in Spring Azure qPID JMS code
问题描述
我正在尝试在我们非常小的和简单的Spring Boot应用程序中跟踪和识别导致内存泄漏的根本原因.
Im trying to trace and identify root cause for memory leak in our very small and simple Spring Boot application.
它使用以下内容: -Spring Boot 2.2.4 -azure-servicebus-jms-spring-boot-starter 2.2.1 -MSSQL
It uses following: - Spring Boot 2.2.4 - azure-servicebus-jms-spring-boot-starter 2.2.1 - MSSQL
功能: 该应用程序仅调度Azure ServiceBus队列并存储数据，并将数据发送到其他目标. 这是一个小型应用程序，因此尽管我通过Xmx选项最多提供了256兆的内存，但它还是很容易以64兆的内存开始.重要说明是使用专用JmsTransactionManager的Spring默认事务处理模式调度队列，该JmsTransactionManager实际上是ChainedTransactionManager的内部TM以及dbTM和其他出站JMS TM.这两个JMS ConnectionFactory对象都创建为CachingConnectionFactory.
Function: The app only dispatches Azure ServiceBus queue and stores data and sends data to other destination. It is a small app so it starts easily with 64 megs of memory, despite I give it up to 256 megs via Xmx option. Important note is the queue is being dispatched using Spring default transacted mode with dedicated JmsTransactionManager who is actually inner TM of ChainedTransactionManager along with dbTM and additional outbound JMS TM. Both JMS ConnectionFactory objects are created as CachingConnectionFactory.
行为:
一旦启动应用程序，看起来就可以了.没有流量，因此我在日志中可以看到它正在打开事务并在检查队列时关闭(jms:message-driven-channel-adapter).
Once the app is started it seems OK. There is no traffic so I can see in the log it is opening transactions and closing when checking the queue (jms:message-driven-channel-adapter).
但是一段时间后仍然没有流量，没有消息被占用，内存开始通过JVVM进行监视.
However after some time when there is still no traffic, no single message was consumed the memory starts climbing as monitored via JVVM.
抛出错误:
--2020-04-24 11:17:01.443 - WARN 39892 --- [er.container-10] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'MY QUEUE NAME HERE' - trying to recover. Cause: Heuristic completion: outcome state is rolled back; nested exception is org.springframework.transaction.TransactionSystemException: Could not commit JMS transaction; nested exception is javax.jms.IllegalStateException: The Session was closed due to an unrecoverable error.
...，几分钟后达到堆的最大值，此后由于打开JMS连接的线程中的OutOfMemory错误而失败.
... and after several minutes it reaches MAX of the heap and since that moment it is failing on OutOfMemory error in the thread opening JMS connections.
--2020-04-24 11:20:04.564 - WARN 39892 --- [windows.net:-1]] i.n.u.concurrent.AbstractEventExecutor : A task raised an exception. Task: org.apache.qpid.jms.provider.amqp.AmqpProvider$$Lambda$871/0x000000080199f840@1ed8f2b9
-
java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
at org.apache.qpid.proton.engine.impl.ByteBufferUtils.newWriteableBuffer(ByteBufferUtils.java:99)
at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.init_buffers(TransportOutputAdaptor.java:108)
at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:56)
at org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.pending(SaslImpl.java:842)
at org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.pending(HandshakeSniffingTransportWrapper.java:138)
at org.apache.qpid.proton.engine.impl.TransportImpl.pending(TransportImpl.java:1577)
at org.apache.qpid.proton.engine.impl.TransportImpl.getOutputBuffer(TransportImpl.java:1526)
at org.apache.qpid.jms.provider.amqp.AmqpProvider.pumpToProtonTransport(AmqpProvider.java:994)
at org.apache.qpid.jms.provider.amqp.AmqpProvider.pumpToProtonTransport(AmqpProvider.java:985)
at org.apache.qpid.jms.provider.amqp.AmqpProvider.lambda$close$3(AmqpProvider.java:351)
at org.apache.qpid.jms.provider.amqp.AmqpProvider$$Lambda$871/0x000000080199f840.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:835)
堆转储:
在整个过程中，我拍摄了几个堆快照，并查看了增加的内容. 我可以看到可疑数量的ConcurrentHashMap/String/Byte []对象.
I took couple of heap snapshots during this whole process and looked at what gets increased. I can see suspicious amount of ConcurrentHashMap/String/Byte[] objects.
有人在此设置和库中有什么线索/提示吗:Spring Boot，Azure JMS依赖项下使用的Apache qPid等?非常感谢.
Has anyone some clue/hint what can be wrong in this setup and libs: Spring Boot, Apache qPid used under the hood of the Azure JMS dependency etc.? Many thanks.
更新#1 我有明确的证据表明问题出在Spring或Azure服务总线启动程序库中-不是自动使用qPid客户端. 我想说的是，库是Bug而不是Spring，只是我的猜测.这是失败的设置的样子:
Update #1 I have clear evidence that the problem is either in Spring or azure service bus starter library - not automatically qPid client used. I would say the library has the bug rather than Spring, just my guess. This is how the failing setup looks like:
- 有两个JMS目标和一个DB，每个目标都有其事务管理器
- 三个TM上方都包裹着ChainedTransactionManager.
- Spring集成应用程序，该应用程序通过jms:message-driven-channel-adapter连接到Azure ServiceBus队列并在该组件上设置事务管理器(如第2点所创建)
- 启动应用程序.无需排队，在10分钟后，应用程序将因OutOfMemoryError而崩溃...在那10分钟内，我观察到调试级别的日志，唯一正在发生的事情是打开和关闭事务使用ChainedTransactionManager ...也如注释中所述，另一个重要条件是第三个JMS TransactionManager ...具有2个TM且稳定，具有3个TM将会崩溃...
推荐答案
进行的其他研究和步骤确定了Spring CachingConnectionFactory 类的最可能根本原因.一旦删除了该内容并仅使用本机类型，问题就消失了，内存消耗状况也变得非常不同且健康.
Additional research and steps taken identified the most likely root cause Spring CachingConnectionFactory class. Once I removed that and used only native types the problem went away and memory consumption profile is very different and healthy.
我不得不说我使用标准构造函数创建了 CachingConnectionFactory ，并且没有进一步配置行为.但是，根据我的经验，这些Spring默认值显然会导致内存泄漏.
I have to say I created CachingConnectionFactory using standard constructor and didnt further configure the behavior. However these Spring defaults clearly lead to memory leak as per my experience.
过去我使用ActiveMq发生内存泄漏，必须使用 CachingConnectionFactory 解决，现在使用 CachingConnectionFactory 时我使用Azure ServiceBus发生内存泄漏..奇怪:)在这两种情况下，我都将其视为错误，因为无论是否涉及缓存，内存管理都应该正确.
In past I had memory leak with ActiveMq which had to be resolved by using CachingConnectionFactory and now I have memory leak with Azure ServiceBus when using CachingConnectionFactory .. strange :) In both cases I see that as bugs because memory management should be correct regardless caching involved or not.
将此标记为我的答案.
Marking this as my answer.
经过测试的情况:当使用自己的TM和两个JMS connectionFactories都类型CachedConnectionFactory接收和发送消息时，会出现问题.最后，我测试了该应用程序.具有CachedConnectionFactory类型的入站连接工厂，而出站只是本机类型...也没有内存泄漏.
Tested case: The problem occurs when receiving and sending message both with its own TM and both JMS connectionFactories are type CachedConnectionFactory. At the end I tested the app. with inbound connection factory of type CachedConnectionFactory and outbound just native type ... no memory leak as well.
这篇关于在Spring Azure qPID JMS代码中跟踪内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
