节点js-连接池的http.request()问题 [英] Node js - http.request() problems with connection pooling

问题描述

考虑以下简单的Node.js应用程序:

Consider the following simple Node.js application:

var http = require('http');
http.createServer(function() { }).listen(8124); // Prevent process shutting down

var requestNo = 1;
var maxRequests = 2000;

function requestTest() {
    http.request({ host: 'www.google.com', method: 'GET' }, function(res) {
        console.log('Completed ' + (requestNo++));

        if (requestNo <= maxRequests) {
            requestTest();
        }
    }).end();
}

requestTest();

它向google.com发出2000个HTTP请求，一个接一个.问题在于它会请求5号并暂停大约3分钟，然后继续处理请求6-10，然后再暂停3分钟，然后再请求11-15，然后暂停，依此类推. 我尝试将www.google.com更改为localhost，这是一个运行我的计算机并返回"Hello world"的极其基本的Node.js应用，但我仍然可以暂停3分钟. em>

It makes 2000 HTTP requests to google.com, one after the other. The problem is it gets to request No. 5 and pauses for about 3 mins, then continues processing requests 6 - 10, then pauses for another 3 minutes, then requests 11 - 15, pauses, and so on. I tried changing www.google.com to localhost, an extremely basic Node.js app running my machine that returns "Hello world", I still get the 3 minute pause.

现在我读到我可以增加连接池限制:

Now I read I can increase the connection pool limit:

http.globalAgent.maxSockets = 20;

现在，如果我运行它，它将处理请求1-20，然后暂停3分钟，然后请求21-40，然后暂停，依此类推.

Now if I run it, it processes requests 1 - 20, then pauses for 3 mins, then requests 21 - 40, then pauses, and so on.

最后，经过一番研究，我了解到可以通过在请求选项中设置agent: false来完全禁用连接池:

Finally, after a bit of research, I learned I could disable connection pooling entirely by setting agent: false in the request options:

http.request({ host: 'www.google.com', method: 'GET', agent: false }, function(res) {
    ...snip....

...它可以正常处理所有2000个请求.

...and it'll run through all 2000 requests just fine.

我的问题，这样做是个好主意吗?是否有可能导致HTTP连接过多的危险?而且为什么它会暂停3分钟，如果我完成了连接，它肯定会直接将其重新添加到池中，以备下一个请求使用，所以为什么要等待3分钟?原谅我的无知.

My question, is it a good idea to do this? Is there a danger that I could end up with too many HTTP connections? And why does it pause for 3 mins, surely if I've finished with the connection it should add it straight back into the pool ready for the next request to use, so why is it waiting 3 mins? Forgive my ignorance.

如果失败了，那么Node.js应用程序发出大量HTTP请求而不锁定或崩溃的最佳策略是什么?

Failing that, what is the best strategy for a Node.js app making a potentially large number HTTP requests, without locking up, or crashing?

我正在Mac OSX 10.8.2上运行Node.js版本0.10.

I'm running Node.js version 0.10 on Mac OSX 10.8.2.

我发现，如果将上面的代码转换为for循环并尝试同时建立一堆连接，则在大约242个连接后我开始出现错误.错误是:

I've found if I convert the above code into a for loop and try to establish a bunch of connections at the same time, I start getting errors after about 242 connections. The error is:

Error was thrown: connect EMFILE
(libuv) Failed to create kqueue (24)

...和代码...

...and the code...

for (var i = 1; i <= 2000; i++) {
    (function(requestNo) {
        var request = http.request({ host: 'www.google.com', method: 'GET', agent: false }, function(res) {
            console.log('Completed ' + requestNo);
        });

        request.on('error', function(e) {
            console.log(e.name + ' was thrown: ' + e.message);
        });

        request.end();
    })(i);
}

我不知道一个负载很重的Node.js应用程序是否可以同时达到这么多同时连接.

I don't know if a heavily loaded Node.js app could ever reach that many simultaneous connections.

推荐答案

您必须消耗响应.

请记住，在v0.10中，我们进入了stream2.这意味着data事件在您开始寻找之前不会发生.因此，您可以执行以下操作:

Remember, in v0.10, we landed streams2. That means that data events don't happen until you start looking for them. So, you can do stuff like this:

http.createServer(function(req, res) {
  // this does some I/O, async
  // in 0.8, you'd lose data chunks, or even the 'end' event!
  lookUpSessionInDb(req, function(er, session) {
    if (er) {
      res.statusCode = 500;
      res.end("oopsie");
    } else {
      // no data lost
      req.on('data', handleUpload);
      // end event didn't fire while we were looking it up
      req.on('end', function() {
        res.end('ok, got your stuff');
      });
    }
  });
});

但是，在不读取数据时不会丢失数据的流的反面是，如果您不读取数据，它们 实际上不会丢失数据！也就是说，它们开始暂停，您必须阅读它们才能取出任何东西.

However, the flip side of streams that don't lose data when you're not reading it, is that they actually don't lose data if you're not reading it! That is, they start out paused, and you have to read them to get anything out.

因此，您的测试中发生的事情是您发出了一堆请求并且不使用响应，然后最终套接字被Google杀死，因为什么也没发生，它假设你死了.

So, what's happening in your test is that you're making a bunch of requests and not consuming the responses, and then eventually the socket gets killed by google because nothing is happening, and it assumes you've died.

在某些情况下，使用不可能消耗传入的消息:也就是说，如果您不对请求添加response事件处理程序，或者完全编写并完成了服务器上的response消息，而从未读取该请求.在这种情况下，我们只是为您转储垃圾中的数据.

There are some cases where it's impossible to consume the incoming message: that is, if you don't add a response event handler on a requests, or where you completely write and finish the response message on a server without ever reading the request. In those cases, we just dump the data in the garbage for you.

但是，如果您正在监听'response'事件，则有责任处理该对象.在第一个示例中添加response.resume()，您会看到它以合理的速度进行.

However, if you are listening to the 'response' event, it's your responsibility to handle the object. Add a response.resume() in your first example, and you'll see it processes on through at a reasonable pace.

这篇关于节点js-连接池的http.request()问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！