Go http.Get，并发和“由对等连接重置” [英] Go http.Get, concurrency, and "Connection reset by peer"

问题描述

我有从一个服务器1000-2000网页下载，我使用go程序和渠道实现高效率。问题是，每次我运行我的程序多达400个请求失败，错误连接复位由对等。

我能做些什么来防止这种情况？

<有趣的一件事是，当我在一个服务器上运行这个程序在同一个国家的网站托管的服务器，0请求失败，所以我猜，有一些延迟的问题（因为它现在运行在不同大陆的服务器上）。

我使用的代码基本上只是一个简单的http.Get（url）请求，没有额外的参数或自定义客户端。

解决方案

由对等连接重置的连接表示远程服务器已发送 RST 强制关闭连接，故意作为限制连接的机制或由于缺少资源。无论哪种方式，您可能打开太多的连接，或重新连接太快。

启动1000-2000个并行连接很少是下载多个页面的最有效的方法，特别是如果大部分或全部来自单个服务器。如果你测试吞吐量，你会发现一个更低的并发级别。

你还要设置 Transport.MaxIdleConnsPerHost 以匹配您的并发级别。如果 MaxIdleConnsPerHost 低于预期的并发连接数，服务器连接将在请求后关闭，只是立即再次打开 - 这将显着减慢您的进度并可能达到服务器施加的连接限制。

I have between 1000-2000 webpages to download from one server, and I am using go routines and channels to achieve a high efficiency. The problem is that every time I run my program up to 400 requests fail with the error "connection reset by peer". Rarely (maybe 1 out of 10 times), no requests fail.

What can I do to prevent this?

One thing that is interesting is that when I ran this program on a server in the same country as the server the website is hosted in, 0 requests failed, so I am guessing there is some problem with delay (as it is now running on a server on a different continent).

The code I am using is basically just a simple http.Get(url) request, no extra parameters or a custom client.

解决方案

The message connection reset by peer indicates that the remote server sent an RST to forcefully close the connection, either deliberately as a mechanism to limit connections, or as a result of a lack of resources. Either way you are likely opening too many connections, or reconnecting too fast.

Starting 1000-2000 connections is parallel is rarely the most efficient way to download that many pages, especially if most or all are coming from a single server. If you test the throughput you will find an optimal concurrency level that is far lower.

You will also want to set the Transport.MaxIdleConnsPerHost to match your level of concurrency. If MaxIdleConnsPerHost is lower than the expected number of concurrent connections, the server connections will often be closed after a request, only to be immediately opened again -- this will slow your progress significantly and possibly reach connection limits imposed by the server.

这篇关于Go http.Get，并发和“由对等连接重置”的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！