去http.Get、并发和“连接重置由对等"； [英] Go http.Get, concurrency, and "Connection reset by peer"

问题描述

我有 1000-2000 个网页要从一台服务器下载，我正在使用 goroutines 和 channels 来实现高效率.问题是每次我运行我的程序时，多达 400 个请求都会失败，并显示错误连接重置由对等方".很少(可能是十分之一)，没有请求失败.

I have between 1000-2000 webpages to download from one server, and I am using go routines and channels to achieve a high efficiency. The problem is that every time I run my program up to 400 requests fail with the error "connection reset by peer". Rarely (maybe 1 out of 10 times), no requests fail.

我能做些什么来防止这种情况发生?

What can I do to prevent this?

有趣的一件事是，当我在与托管网站的服务器位于同一国家/地区的服务器上运行此程序时，有 0 个请求失败，因此我猜测延迟存在一些问题(就像现在一样)在不同大陆的服务器上运行).

One thing that is interesting is that when I ran this program on a server in the same country as the server the website is hosted in, 0 requests failed, so I am guessing there is some problem with delay (as it is now running on a server on a different continent).

我使用的代码基本上只是一个简单的 http.Get(url) 请求，没有额外的参数或自定义客户端.

The code I am using is basically just a simple http.Get(url) request, no extra parameters or a custom client.

推荐答案

connection reset by peer 消息表示远程服务器发送了 RST 强制关闭连接，或者故意作为限制连接的机制，或者作为缺乏资源的结果.无论哪种方式，您都可能打开太多连接，或者重新连接太快.

The message connection reset by peer indicates that the remote server sent an RST to forcefully close the connection, either deliberately as a mechanism to limit connections, or as a result of a lack of resources. Either way you are likely opening too many connections, or reconnecting too fast.

并行启动 1000-2000 个连接很少是下载这么多页面的最有效方式，尤其是当大部分或全部来自单个服务器时.如果您测试吞吐量，您会发现最佳并发级别要低得多.

Starting 1000-2000 connections in parallel is rarely the most efficient way to download that many pages, especially if most or all are coming from a single server. If you test the throughput you will find an optimal concurrency level that is far lower.

您还需要设置 Transport.MaxIdleConnsPerHost 以匹配您的并发级别.如果 MaxIdleConnsPerHost 低于预期的并发连接数，服务器连接通常会在请求后关闭，然后立即再次打开——这将显着减慢您的进度，并可能达到强加的连接限制由服务器.

You will also want to set the Transport.MaxIdleConnsPerHost to match your level of concurrency. If MaxIdleConnsPerHost is lower than the expected number of concurrent connections, the server connections will often be closed after a request, only to be immediately opened again -- this will slow your progress significantly and possibly reach connection limits imposed by the server.

这篇关于去http.Get、并发和“连接重置由对等"；的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！