每个代理而不是每个路由的Apache HttpClient 4持久连接 [英] Apache HttpClient 4 persistent connection per Proxy instead of per route

问题描述

我的理解是，ClientConnectionManager的所有实现都基于路由持久连接.如果涉及代理，则基本上不会导致持久连接.例如，HttpClient需要通过具有固定IP的HTTP代理访问1000个不同的域，它必须与代理建立至少1000个连接，而不是与代理建立1个持久连接并为1000个请求重复使用.

我正在模拟多个用户访问数千个域(虚假域，所有dns解析为几个IP，解析发生在代理之后，因此与HttpClient无关).当我增加用户和域的数量时，上述行为很快耗尽了localhost中所有可用的端口，结果发生了地址绑定错误.

有没有一种方法可以使HttpClient保持基于代理的连接? IE. HttpClient仅维护与给定代理的指定数量的连接.

解决方案

经过深入研究，Apache HttpClient似乎不支持这种行为.我必须修改HttpClient/HttpCore源才能具有此功能，即.维护仅基于localAddress和第一个代理地址的持久连接.

我修改的类是:

org.apache.http.conn.routing.HttpRounte.java和 org.apache.http.conn.routing.BasicRouteDirector.java.

基本上，我在HttpRoute中更改了hashCode和equal方法(用作持久conn查找的哈希表的键)，因此如果涉及代理，则查找不考虑目标地址.

在我的场景中，以上修改的初始测试结果显示，请求吞吐量提高了约100倍.到目前为止，它对我来说还不错.

凯文

My understanding, all implementations of ClientConnectionManager persist connections base on route. This results in basically no persistent connections if a proxy is involved. For example, the HttpClient needs to visit 1000 different domains via a HTTP proxy with an fix IP, it has to establish at least 1000 connection to the proxy instead of creating 1 persistent connection to the proxy and reuse that for the 1000 requests.

I'm simulating multiple users visiting thousands of domains (fake domains, all dns resolved to a couple of IPs, the resolving happen after the proxy, so nothing to do with HttpClient). The above behavior quickly use up all available ports in the localhost as I increase the # of users and domains, the Address Bind errors occur as result.

Is there a way to make the HttpClient to persist connection on proxy basis? ie. A HttpClient only maintain specified number of connections to a given proxy.

解决方案

After intensive research, it seems that Apache HttpClient doesn't support this behavior out-of-box. I have to modify the HttpClient/HttpCore source in order to have this feature, ie. maintain persistent connections based only on localAddress and First Proxy address.

The classes I modified are:

org.apache.http.conn.routing.HttpRounte.java and org.apache.http.conn.routing.BasicRouteDirector.java.

Basically I changed the hashCode and equal method in HttpRoute (which is used as a key to hashtable for persistent conn lookup), so the lookup doesn't consider target address if a proxy is involved.

Initial test results of above modification shows about 100 times improvement in terms of request throughput in my scenario. So far it works fine for me.

Kevin

这篇关于每个代理而不是每个路由的Apache HttpClient 4持久连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！