如果一个线程调用getStackTrace()，而在另一个线程中发生lambda定义(通过Unsafe.defineAnonymousClass)，则Java8会挂断 [英] Java8 hangs up if getStackTrace() is called by one thread and lambda definition (via Unsafe.defineAnonymousClass) occurs in another thread

问题描述

我的Web应用程序在16核(带有HT的32-核)Dual-Xeon NUMA计算机上以JVM 1.8.0_45-b15和Windows Server 2012运行在Apache Tomcat/8.0.21中，可能会卡在在某些非常不幸的情况下，标题中描述的操作在两个不同的线程中同时发生.

My Web application, which runs in Apache Tomcat/8.0.21, with JVM 1.8.0_45-b15 and Windows Server 2012 on a 16-core (32- with HT) Dual-Xeon NUMA machine, can get stuck, in some very unfortunate circumstances, when the actions described in the title occur at the same time in two different threads.

执行第一个动作(getStackTrace())的线程正在尝试执行一些诊断，以检测系统的哪部分正在减慢速度并在调用Thread.dumpThreads时卡住. 另一个线程正在执行某些操作，其中包括部分JVM的内部lambda定义.

The thread doing the first action (getStackTrace()) is trying to perform some diagnostic to detect which part of the system is slowing things down and gets stuck while calling Thread.dumpThreads. The other thread is doing some operation, which includes under-the-hood lambda definition on part of the JVM.

特别是，我具有以下堆栈跟踪(通过jstack -F <pid>获得):

In particular, I have the following stack trace (obtained with jstack -F <pid>):

Attaching to process ID 6568, please wait... 
Debugger attached successfully. 
Server compiler detected. 
JVM version is 25.45-b02 

Deadlock Detection:

No deadlocks found. (... well, that's not the kind of deadlock you were searching for, dear JVM, but something bad is happening altogether :( )

Thread 155: (state = BLOCKED)
 - sun.misc.Unsafe.defineAnonymousClass(java.lang.Class, byte[], java.lang.Object[]) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.invoke.InvokerBytecodeGenerator.loadAndInitializeInvokerClass(byte[], java.lang.Object[]) @bci=8 (Compiled frame)
 - java.lang.invoke.InvokerBytecodeGenerator.loadMethod(byte[]) @bci=6 (Compiled frame)
 - java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(java.lang.invoke.LambdaForm, java.lang.invoke.MethodType) @bci=17 (Compiled frame)
 - java.lang.invoke.LambdaForm.compileToBytecode() @bci=65 (Compiled frame)
 - java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(java.lang.invoke.MethodType, int) @bci=638 (Interpreted frame)
 - java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.lang.invoke.MethodType, int) @bci=17 (Compiled frame)
 - java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.lang.invoke.MemberName) @bci=163 (Compiled frame)
 - java.lang.invoke.DirectMethodHandle.make(byte, java.lang.Class, java.lang.invoke.MemberName) @bci=94 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(byte, java.lang.Class, java.lang.invoke.MemberName, boolean, boolean, java.lang.Class) @bci=201 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(byte, java.lang.Class, java.lang.invoke.MemberName, java.lang.Class) @bci=8 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(byte, java.lang.Class, java.lang.invoke.MemberName) @bci=30 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(byte, java.lang.Class, java.lang.String, java.lang.Object) @bci=115 (Compiled frame)
 - java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(java.lang.Class, int, java.lang.Class, java.lang.String, java.lang.Object) @bci=38 (Compiled frame)
 - c.e.s.w.t.si.a.DDVP.lambda$1(com.vaadin.data.Container, com.vaadin.ui.HorizontalLayout, com.vaadin.ui.Label, com.vaadin.ui.Label, java.lang.String, java.lang.String, java.util.Map) @bci=48, line=104 (Interpreted frame)
 - c.e.s.w.t.si.a.DDVP$$Lambda$637.updateUIWith(java.lang.Object) @bci=32 (Interpreted frame)
 - c.e.s.w.d.DU$VoidUIUpdaterFromUIUpdater.updateUI() @bci=8, line=321 (Compiled frame)
 - c.e.s.w.d.DU$CompletionSignallingVoidUIUpdater.updateUI() @bci=4, line=125 (Compiled frame)
 - c.e.s.w.d.CUQ$1.sweepWhileNotTimedOut() @bci=59, line=218 (Compiled frame)
 - c.e.s.w.d.CUQ$QueueExhauster.run() @bci=247, line=122 (Compiled frame)
 - c.e.s.w.d.CUQ$DequeuerStartFailed.run() @bci=40, line=60 (Compiled frame)
 - c.e.s.w.s.ew.CC.lambda$4(java.lang.Runnable) @bci=13, line=66 (Compiled frame)
 - c.e.s.w.s.ew.CC$$Lambda$59.run() @bci=8 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
 - java.lang.Thread.run() @bci=11 (Compiled frame)

Thread 108: (state = BLOCKED) [The tricky one...]
 - java.lang.Thread.dumpThreads(java.lang.Thread[]) @bci=0 (Interpreted frame)
 - java.lang.Thread.getStackTrace() @bci=41 (Compiled frame)
 - c.e.s.w.SWA$$Lambda$98.getStackTrace() @bci=4 (Interpreted frame)
 - c.e.s.w.SWA.describeHoggingCode(c.e.s.w.s.RequestTimeTracker$StackTraceProvider, boolean) @bci=1, line=401 (Interpreted frame)
 - c.e.s.w.SWA.describeHoggingCode(c.e.s.w.s.RequestTimeTracker$StackTraceProvider, boolean, java.lang.Thread) @bci=6, line=396 (Interpreted frame)
 - c.e.s.w.SWA.lambda$10(java.lang.Thread, java.lang.String) @bci=8, line=890 (Interpreted frame)
 - c.e.s.w.SWA$$Lambda$62.run() @bci=12 (Interpreted frame)
 - c.e.s.w.s.ew.CC.lambda$4(java.lang.Runnable) @bci=13, line=66 (Compiled frame)
 - c.e.s.w.s.ew.CC$$Lambda$59.run() @bci=8 (Compiled frame)
 - c.e.s.w.s.IS.lambda$4(java.util.concurrent.atomic.AtomicBoolean, java.lang.Runnable) @bci=8, line=327 (Compiled frame)
 - c.e.s.w.s.IS$$Lambda$60.run() @bci=8 (Compiled frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4 (Compiled frame)
 - java.util.concurrent.FutureTask.run() @bci=42 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=30 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
 - java.lang.Thread.run() @bci=11 (Interpreted frame)

从我的角度来看，此问题可能与Unsafe.defineAnonymousClass无法处理正在进行的对java.lang.Thread.dumpThreads的调用有关(反过来，这需要在JVM中实现java.lang.Thread.getStackTrace).关键是，由于final或package修饰符，我无法扩展此过程中涉及的任何核心类(例如Lookup，MethodHandleNatives等)以引入锁当仍在呼叫java.lang.Thread.dumpThreads时，这将阻止棘手的不安全呼叫.另外，我怀疑引入这样的锁定还会使速度降低很多，因为，到处都是lambda.

From my point of view, this issue could be related to Unsafe.defineAnonymousClass not being able to cope with an ongoing call to java.lang.Thread.dumpThreads (which, in turn, is needed to implement java.lang.Thread.getStackTrace, inside JVM). The key point is that, due final or package modifiers, I'm not able to extend any of the core classes involved in this process (such as Lookup, MethodHandleNatives, etc.) in order to introduce a lock which would be blocking tricky Unsafe calls while a call to java.lang.Thread.dumpThreads is still ongoing. Also, I suspect that introducing such lock could also slow things down quite a bit, since, well, lambdas are everywhere.

有人遇到过类似的问题吗?您能帮忙解决吗?

Has anyone encountered a similar problem? Could you help solving it?

谢谢！

当然，在堆栈跟踪中也有类似这样的线程，我将其省略，因为我认为它们与这种情况无关.

Of course, in the stacktrace were also threads like these, which I omitted since I think they are not relevant to this case.

Thread 154: (state = BLOCKED) [Many of these....]
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78 (Compiled frame)
 - org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=57, line=389 (Compiled frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=516 (Compiled frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=47 (Compiled frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=300, line=575 (Compiled frame)
 - java.lang.Thread.run() @bci=11 (Compiled frame)


Thread 153: (state = BLOCKED) [and of these...]
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=354 (Compiled frame)
 - java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44 (Interpreted frame)
 - java.util.concurrent.ForkJoinWorkerThread.run() @bci=24 (Interpreted frame)


Thread 141: (state = BLOCKED) [and of these...]
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11 (Interpreted frame)

推荐答案

最后，我设法解决了这个令人讨厌的问题.如果选项

Finally I managed to solve this nasty issue. It appears that JVM can get stuck with traces like the ones if posted (involving at least sun.misc.Unsafe.defineAnonymousClass) if the option -XX:-ClassUnloading is passed on to the JVM executable (as was the case for the JVM I was trying to cure). For some reason this can stimulate this deadlock-style bug in Java system libraries, which appears to be random (as all deadlock bugs) and getting more and more likely as JVM process "ages". Again, this is nothing new for deadlock bugs: the more "gears" (say, classes which should had been unloaded but weren't due to that option) you put into the machine, the more likely is that two or more of them crank.

因此，删除选项-XX:-ClassUnloading会使此问题完全消失.

As a result, removing the option -XX:-ClassUnloading makes this problem completely disappear.

最重要的是:切勿在生产系统中使用-XX:-ClassUnloading(或让任何人在JVM进程启动脚本中放置这样的选项)，甚至在

The bottom line is: never use -XX:-ClassUnloading in production systems (or let anyone put such option in JVM process starting scripts), not even in Java8, which shouldn't be limited by PermGen, yet it still can suffer badly due to problem in sun.misc.Unsafe.defineAnonymousClass .

这篇关于如果一个线程调用getStackTrace()，而在另一个线程中发生lambda定义(通过Unsafe.defineAnonymousClass)，则Java8会挂断的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！