spray-client抛出“打开的文件太多"发出更多并发请求时发生异常 [英] spray-client throwing "Too many open files" exception when giving more concurrent requests

问题描述

我有一个运行在服务器X中的Spray http客户端，它将连接到服务器Y.服务器Y有点慢(请求需要3秒钟以上的时间)

I have a spray http client which is running in a server X, which will make connections to server Y. Server Y is kind of slow(will take 3+ sec for a request)

这是我的http客户端代码调用:

This is my http client code invocation:

def get() {
    val result = for {
       response <- IO(Http).ask(HttpRequest(GET,Uri(getUri(msg)),headers)).mapTo[HttpResponse]
    } yield response

    result onComplete {
      case Success(res) => sendSuccess(res)
      case Failure(error) => sendError(res)
    }
}

这些是我在application.conf中拥有的配置:

These are the configurations I have in application.conf:

spray.can {
    client {
            request-timeout = 30s
            response-chunk-aggregation-limit = 0
            max-connections = 50
            warn-on-illegal-headers = off
        }
        host-connector {
            max-connections = 128
            idle-timeout = 3s
          }
    }

现在，我尝试滥用具有大量并发请求的服务器X(使用n = 1000和c = 100的ab).

Now I tried to abuse the server X with large number of concurrent requests(using ab with n=1000 and c=100).

直到900请求一切正常.在那之后，服务器抛出了很多异常，在那之后我打不开服务器. 这些是例外:

Till 900 requests it went fine. After that the server threw lot of exceptions and I couldn't hit the server after that. These are the exceptions:

[信息] [错误] [2015年3月28日17:33:13.276] [squbs-akka.actor.default-dispatcher-6] [akka://squbs/system/IO-TCP/selectors/$ a/0]接受错误:无法接受新连接

[info] [ERROR] [03/28/2015 17:33:13.276] [squbs-akka.actor.default-dispatcher-6] [akka://squbs/system/IO-TCP/selectors/$a/0] Accept error: could not accept new connection

[info] java.io.IOException:打开的文件太多 sun.nio.ch.ServerSocketChannelImpl.accept0上的[info](本机方法) [info]位于sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) [info]位于akka.io.TcpListener.acceptAllPending(TcpListener.scala:103)

[info] java.io.IOException: Too many open files [info] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) [info] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) [info] at akka.io.TcpListener.acceptAllPending(TcpListener.scala:103)

并进一步击中同一台服务器时，它引发了以下异常:

and on further hitting the same server, it threw the below exception:

[信息] [错误] [2015年3月28日17:53:16.735] [hcp-client-akka.actor.default-dispatcher-6] [akka://hcp-client/system/IO-TCP/selectors] null [info] akka.actor.ActorInitializationException:创建期间发生异常

[info] [ERROR] [03/28/2015 17:53:16.735] [hcp-client-akka.actor.default-dispatcher-6] [akka://hcp-client/system/IO-TCP/selectors] null [info] akka.actor.ActorInitializationException: exception during creation

[info] atkka.actor.ActorInitializationException $ .apply(Actor.scala:164)

[info] at akka.actor.ActorInitializationException$.apply(Actor.scala:164)

[info] atkka.actor.ActorCell.create(ActorCell.scala:596)

[info] at akka.actor.ActorCell.create(ActorCell.scala:596)

[info]由:java.lang.reflect.InvocationTargetException

[info] Caused by: java.lang.reflect.InvocationTargetException

[info]位于sun.reflect.GeneratedConstructorAccessor59.newInstance(未知来源)

[info] at sun.reflect.GeneratedConstructorAccessor59.newInstance(Unknown Source)

[info]原因:java.io.IOException:打开的文件太多 [info]位于sun.nio.ch.IOUtil.makePipe(本机方法)

[info] Caused by: java.io.IOException: Too many open files [info] at sun.nio.ch.IOUtil.makePipe(Native Method)

我以前使用的是apache http客户端(同步的)，该客户端能够处理10000+并发100的请求.

I was previously using apache http client(which was synchronous) which was able to handle 10000+ requests with concurrency of 100.

我不确定我缺少什么.任何帮助将不胜感激.

I'm not sure I'm missing something. Any help would be appreciated.

推荐答案

问题在于，每次调用get()方法时，它都会创建一个新的actor，该actor创建与远程服务器的至少一个连接.此外，您永远不会关闭该参与者，因此每个这样的连接都会离开，直到超时为止.

The problem is that every time you call get() method it creates a new actor that creates at least one connection to the remote server. Furthermore you never shut down that actor, so each such connection leaves until it times out.

您只需要一个这样的角色即可管理所有HTTP请求，因此要解决此问题，可以将IO(Http)从get()方法中删除，并仅调用一次.对于返回到该服务器的所有请求，重复使用返回ActorRef的形式.在应用程序关闭时将其关闭.

You only need a single such actor to manage all your HTTP requests, thus to fix it take IO(Http) out of the get() method and call it only once. Reuse that returned ActorRef for all your requests to that server. Shut it down on application shutdown.

例如:

val system: ActorSystem = ...
val io = IO(Http)(system)
io ! Http.Bind( ...

def get(): Unit = {
  ...
  io.ask ...
  // or
  io.tell ...
}

这篇关于spray-client抛出“打开的文件太多"发出更多并发请求时发生异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！