在Jetty HttpClient Hang上寻求建议 [英] Seeking advice on Jetty HttpClient Hang
问题描述
我有一个小型应用程序，可以使用Jetty v9.2 HttpClient 来轮询服务器.几天后，该应用程序将冻结.最初，我们确定线程池需要为
I have a small application that simply polls a server using Jetty v9.2, HttpClient. After some days, the application will freeze-up. Initially we identified the thread pool needed to be increased in size to relieve a performance hit. That change restored performance over a period of days. The lock-up remains. The cause has been isolated to the HTTP GET calls (problem goes away if when we comment-out the method).
出现在 Jetty HttpClient 连接管理或线程管理底层的根本原因.通常，Jetty HttpClient会创建一组线程来处理HTTP GET(请参见下文)，这些线程会如您所愿的消失.在运行约40个小时后，JDK VisualVM会显示至少9个连接线程，不不消失会立即消失:
The root cause which appears the underlying the Jetty HttpClient Connection management or Thread management. Normally Jetty HttpClient makes a set of threads to handle the HTTP GET (see below), these run-up and vanish as you'd expect. After around 40 hours or operation, the JDK VisualVM shows at least 9 connection threads that do not go away immediately:
- HttpClient-调度程序x 1
- HttpClient-选择器客户端SeclectorManager x 4
- HttpClient x 4
也
- RMI TCP连接
总共9或10个线程.下次读取时，将创建新的线程实例来承担负载，客户端继续进行.此外，该应用程序.有一个带有专用线程的时钟，该时钟在应用程序锁定后继续运行，这表明JVM，操作系统和机器本身都很好.
Nine or 10 threads in total. On the next read, new thread instances are created to carry the load and the client proceeds. Furthermore the app. has a clock with a dedicated thread which continues running after the the application locks-up, which indicating the JVM, operating system and the machine itself are fine.
有时，我们看到这些卡住"的线程会停留长达一个小时，然后才会退出VisualVM线程显示.至少36个小时后，我们看到线程仍然存在，并且还没有看到它们消失.
Sometimes, we see these 'stuck' threads linger for up to an hour before they drop out of the VisualVM thread display. After at least 36 hours we see threads remain and we've not seen them go away.
足够的时间后，软件将锁定.指示的解释是未清除的线程实例的泄漏.出现该应用程序.线程用完了，无法做更多的工作.正如服务器日志所证实的那样，它肯定会停止HTTP GET.
After enough days the software locks-up. The indicated explanation is the leaking of thread instances that have not been cleaned-up. it appears the app. runs-out of threads and can't do more work. It certainly stops HTTP GETs as witnessed by server-logs.
主要的HTTP调用使用下面的代码HttpClient GET方法:
The main HTTP call uses the code below, HttpClient GET method:
/**
* GET
* @return null or string returned from server
**/
public static String get( final String command ){
String rslt = null;
final String reqStr = "http://www.google.com"; // (any url)
HttpClient httpClient = new HttpClient();
Request request;
ContentResponse response;
try {
//-- Start HttpClient
httpClient.start();
request = httpClient.newRequest( reqStr );
response = request.send();
if( null == response ){
LOG.error( "NULL returned from previous HTTP request.");
}
else {
if( (501 == response.getStatus()) || (502 == response.getStatus()) ){
setNetworkUnavailable(String.format("HTTP Server error: %d", response.getStatus() ));
}
else {
if( 404 == response.getStatus() ){
Util.puts(LOG,"HTTP Server error: 404");
// ignore message since we are talking to an old server
}
else if( 200 == response.getStatus() ){
rslt = response.getContentAsString();
}
else {
LOG.error(String.format( " * Response status: \"%03d\".", response.getStatus() ));
}
setNetworkAvailable();
}
}
}
catch ( InterruptedException iEx ){
LOG.warn( "InterruptException processing: "+reqStr, iEx );
}
catch ( Exception ex ){
Throwable cause = eEx.getCause();
if( (cause instanceof NoRouteToHostException) ||
(cause instanceof EOFException) ||
(cause instanceof SocketException)
&& cause.getMessage().startsWith( EX_NETWORK_UNREACHABLE ) ){
setNetworkUnavailable( cause.getMessage() );
}
else {
LOG.error( "Exception on: "+command, ex );
}
}
finally {
try {
httpClient.stop();
}
catch ( Exception ex ){
LOG.error( "Exception httpClient.stop(), ServerManager::get()", ex );
}
}
return rslt;
}//get method
这是基于简单的示例，有关HttpClient使用的详细信息很少.按照霍伊尔的说法，一切都做完了吗?
This is based on simple examples, there is scant detail on use of the HttpClient. Is everything done according to Hoyle?
在不同的执行运行中，我们还会看到以下异常和日志消息:
At different execution runs we also see the following Exceptions and log messages:
- [36822522] WARN 2014-Sep-02 02:46:28.464> HttpClient @ 2116772232 {STOPPING，8< = 0< = 200，i = 0，q = 0}无法停止线程[HttpClient @ 2116772232- 729770,5，]
我们想知道此消息是否与卡住的线程之一有关?或者，此消息是否表明我们需要研究一个单独且不同的问题?另外:
We wonder if this message relates to one of the stuck threads? Or, does this message indicate a separate and different problem we need to examine? Also:
- java.util.concurrent.TimeoutException(ExecutionException)
这似乎是线程超时异常.哪个线程呢?这与HTTP连接线程有关吗?我认为，当服务在内部捕获错误时，它们至少可以至少指示错误的位置和堆栈跟踪.
This appears to be a thread timeout exception. Which thread though? Does this relate to he HTTP connection-threads? I think as a minimum when services catch errors internally that they can at least indicate the location of the error and a stack-trace.
有一些明显的问题:
- 是否按照要求编写了get()方法代码，以确保Jetty HttpClient代码没有泄漏或没有资源挂起?
- 我们如何捕获警告:"无法停止线程"错误?
- 此错误的影响是什么?有没有办法像这样卡住线程来"粉碎"?
- 这是否与10个悬挂式连接螺纹有关?只有一条警告消息.
- 一个想象中的悬挂线程需要一个ERROR标签，而不是警告.
- Is the get() method code written as required to not have leaks or leave resources hanging for the Jetty HttpClient code?
- How can we catch the warning: "Couldn't stop Thread" error?
- What is the impact of this error? Is there a way to 'smash' a thread stuck like that?
- Does this relate to the 10 hanging connection threads anyway? There's only one warning message.
- One imagine a hanging thread warrants an ERROR label, not a warning.
- 是否可以使用设置直接影响线程锁定?
- 仅在显示的GET方法中进行码头呼叫(尽管有更多日志记录等)
另一个观察结果是，当我们在VisualVM中阻塞"线程时，它在线程"面板中显示了多余的守护程序线程，而不是非守护程序线程的增加.
One other observation is that when we "stuck" threads in VisualVM, it shows excess Daemon threads in the Threads panel, not an increase in non-Daemon threads.
通过在for循环中运行上面显示的代码大约3或4个小时，并且HttpClient send()调用之间的间隔为250毫秒，这表明线程泄漏-在Linux上很容易重现.日志输出显示无警告，并且距线程泄漏至少30分钟距离的网络上只有两个超时错误.
By running the code shown above in a for loop for about 3 or 4 hours with a 250 millisecond break between HttpClient send() calls shows a thread leak -- It is simple to reproduce on Linux. The log output shows no WARNings and only two timeout errors on the network at least 30 minutes distance from the thread leak.
我们非常欢迎您提出建议，观察，改进和回答.预先感谢.
Suggestions, observations, improvements and answers are most welcome. Our thanks in advance.
相关问题:
这些问题涵盖了一些非常相似的观点
These questions cover some very similar points
- Jetty stopping without reason
- ConnectionPoolTimeoutException: timeout waiting for connection from pool
- Unable to shut down Neo4j Jetty server within Tomcat webapp -
推荐答案
这种情况似乎可以通过确保两件事来解决.
This situation seems to be resolved by ensuring two things.
- 确保应用程序的线程池中有足够的线程
- 确保使用Jetty清理代码并捕获/管理所有异常.
这两个动作是相互关联的.如果HttpClient有时会遗漏异常或错误，则线程会四处徘徊.似乎避免这种情况的唯一方法是确保使用的每个HttpCLient调用 HttpCLient.stop().这需要放在最终{...} 子句中.
The two actions are inter-related. If sometimes the HttpClient misses an exception, or error, the thread hangs around. It seems the only way to avoid this is to ensure every HttpCLient used calls HttpCLient.stop(). This needs to go in a finally {...} clause.
第二次异步调用必须等待CompleteListener才能调用HttpCLient.stop().这似乎是确保停工干净"完成的唯一方法.在某些情况下，stop()调用似乎可以继续进行.最终，某些会导致异常，并且您的应用程序会慢慢泄漏资源.外观就像JVM已冻结，但某些非常规任务可能会继续执行(例如GUI线程)，直到PC自身资源耗尽或崩溃，您才可能注意到该问题.这是一个极端的情况，##标题##运行了几个星期.
Secondly async calls must wait for the CompleteListener before calling HttpCLient.stop(). That seems to be the only way to ensure the stop was 'cleanly' done. For some cases, stop() calls appear to proceed OK. Eventually some will cause Exceptions and you application slowly leaks resources. The appearance is like the JVM has frozen, but some non-deamon tasks may continue (e.g. a GUI thread) and you may not notice the problem until the PC itself runs out of resources or crashes. That is an extreme case ## Heading ##running over several weeks.
此处显示了一个适当关闭HttpClient的可靠示例:
A reliable example to appropriately close the HttpClient is shown here:
线程数取决于您的应用程序.我建议使用 jVisualVM 或类似的方法来确保您的码头在调整线程池中的线程数之前，首先必须正确清理所有线程.
The number of threads will depend on your application. I suggest using jVisualVM or something similar to ensure your Jetty threads are all cleaning-up properly first, before tuning the number of threads in your thread pool.
我认为文档需要强调清理工作并确保调用stop().据我所知，有关如何结束Async调用的信息尚未公开.只要您的Jetty呼叫干净地停止，那么似乎就会提供足够的线程来解决此问题-通常需要注意一些并发管理问题.
I feel that the documentation needs to stress the cleaning up and ensuring stop() is called. And the information of how to conclude an Async call is undocumented as far as I can tell. As long as your Jetty calls stop cleanly, then providing sufficient threads appears to resolve this -- With the usual caveats to manage concurrency.
这篇关于在Jetty HttpClient Hang上寻求建议的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
