如何解决“套接字挂起错误"真正快速地发出大量请求时 [英] How to fix the "Socket Hangup Error " when large number of requests are made really quick

问题描述

我有一个nodejs应用程序，该应用程序聚合来自各个网站的内容.发出请求以使用请求流异步地从不同源获取提要.发出请求时，我经常会收到套接字挂断错误.

I have a nodejs application that aggregates contents from various websites. Requests are made to fetch the feeds from different sources asynchronously using request streams. I get the socket hangup error pretty often when the requests are made.

err in accessing the link { Error: socket hang up
    at createHangUpError (_http_client.js:331:15)
    at TLSSocket.socketOnEnd (_http_client.js:423:23)
    at emitNone (events.js:111:20)
    at TLSSocket.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9) code: 'ECONNRESET' } https://arstechnica.com/?p=1488489 

环境详细信息: 节点版本-v8.12.0

Environment details: node version - v8.12.0

尝试了相关SO帖子中给出的一些建议，但是我仍然遇到相同的错误. NodeJS-"socket hang up"是什么意思?真的是什么意思?

Tried out a few suggestions given in related SO posts, but I still get the same error. NodeJS - What does "socket hang up" actually mean?

import request from 'request';
import FeedParser from 'feedparser';

const extractor = require('unfluff');

export const getFeedsFromSource = function (urlfeed, etag, LastModified, callback) {
  console.log(urlfeed, etag, LastModified);
  const req = request({
    method: 'GET',
    url: urlfeed,
    headers: {
      'If-None-Match': etag,
      'If-Modified-Since': LastModified,
      Connection: 'keep-alive',
      ciphers: 'DES-CBC3-SHA',
    },
  });
  const feedparser = new FeedParser();
  const metaData = {};
  const htmlData = {};
  const feedData = {};
  // const pList = null;
  req.on('response', function (response) {
    const stream = this;
    if (response.statusCode === 304) {
      console.log('Source not modified: ', urlfeed);
    }
    if (response.statusCode === 200) {
      metaData.etagin = response.headers.etag;
      metaData.LastModifiedin = response.headers['last-modified'];
      metaData.LastModifiedLocal = response.headers['last-modified'];
      stream.pipe(feedparser).end();
    }
  });
  req.on('error', (err) => {
    console.log(`getFeed: err.message == ${err.message}`);
    callback(err);
  });
  // req.end();
  feedparser.on('readable', function () {
    try {
      const item = this.read();
      if (item !== null) {
        request({
          method: 'GET',
          url: item.link,
        }, (err, info) => {
          if (!err) {
            htmlData.body = info.body;
            const parsedData = extractor(htmlData.body, 'en');
            feedData.author = [];
            feedData.videos = [];
            feedData.feedtitle = parsedData.title;
            feedData.feedmainpicture = parsedData.image;
            feedData.feedsummary = parsedData.description;
            feedData.feedmaincontent = parsedData.text;
            feedData.author.push(item.author);
            if (item.author === null) {
              feedData.author = parsedData.author;
            }
            feedData.feedurl = item.link;
            feedData.copyright = item.meta.copyright;
            // feedData.videos = parsedData.videos;
            feedData.publishedDate = item.pubdate;
            if (item.categories.length > 0) {
              feedData.categories = item.categories;
              feedData.feedtags = item.categories;
            } else if (parsedData.keywords !== undefined) {
              feedData.categories = parsedData.keywords.split(' ').join('').split(',');
              feedData.feedtags = parsedData.keywords.split(' ').join('').split(',');
            } else {
              feedData.categories = [];
              feedData.feedtags = [];
            }
            metaData.sourcename = item.meta.title;
            callback(undefined, feedData, metaData);
          } else {
            console.log('err in accessing the link', err, item.link);
          }
        });
      }
    } catch (err) {
      console.log(`getFeed: err.message == ${err.message}`);
    }
  });
  feedparser.on('error', (err) => {
    console.log(`getFeed: err.message == ${err.message}`);
  });
  feedparser.on('end', () => {
    console.log('onend');
  });
};

请帮助我解决此问题.

Kindly help me out with this issue.

推荐答案

在生产应用中，套接字挂起/重置的原因很多.从您的描述中，我相信原因不是由于应用程序请求超载(除非您运行的计算机非常慢). IMO，最可能的候选者是由于来自同一IP的连接过多而导致远程服务器节流(chrome最多可打开8个连接到任何单个服务器的连接，尽管每个服务器都有不同的限制，您也应尽量不超过此限制)，以解决此问题您应该执行以下操作之一:

There are many reasons for socket hangup/reset in production apps. From your description I believe the cause isn't due to app overloading with requests (unless you're running a very slow machine). IMO, the most likely candidate is throttling by remote server due to too many connections from same ip (chrome opens upto 8 connections to any single server, you should try not to exceed this limit, despite each server having different limit), to solve this you should do one of the following:

• 添加主机请求池(基本设置为Agent.maxSockets)
• 使用代理服务(例如Luminati)在许多源ip上分发请求(与高并发要求更相关)

还有一件要记住的事，由于自然的"网络原因(例如，网络连接不稳定，网络繁忙，服务器繁忙导致峰值)，请求可能会失败，因此在放弃之前，您应始终至少重试一次请求.

One more thing to remember, requests can fail for 'natural' networking reasons (e.g. bad\unstable internet connection, server busy spikes), you should always do at least one retry of request before giving up.

这篇关于如何解决“套接字挂起错误"真正快速地发出大量请求时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！