Golang Http获取请求非常慢 [英] Golang Http Get Request very slow

问题描述

在Golang中使用简单的HTTP Get Request遇到一个非常奇怪的问题.

I have a very strange problem with a simple HTTP Get Request in Golang.

每个Golang中对 https://www.alltron.ch的请求/json/searchSuggestion?searchTerm = notebook 大约需要6-8秒(！)

Every request in Golang to https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook needs about 6-8 seconds (!)

如果在 Chrome，Postman或Powershell 中触发了相同的请求，则所需时间不到一秒钟.

If same request fired in Chrome, with Postman or with Powershell it needs less than a second.

有人知道为什么会这样吗?

Does somebody has a clue why this happens?

我的代码:

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    client := &http.Client{}

    req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)

    response, err := client.Do(req)
    if err != nil && response == nil {
        log.Fatalf("Error on request. %v", err)
    }
    defer response.Body.Close()

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        log.Fatalf("Couldn't get response body. %v", err)
    }

    fmt.Print(string(body))
}

推荐答案

您要访问的网站位于Akamai CDN的后面:

The site you are trying to access is behind the Akamai CDN:

$ dig www.alltron.ch 
...
www.alltron.ch.         152     IN      CNAME   competec.botmanager.edgekey.net.
competec.botmanager.edgekey.net. 7052 IN CNAME  e9179.f.akamaiedge.net.
e9179.f.akamaiedge.net. 162     IN      A       2.20.176.40

Akamai向其客户提供不是浏览器的Web客户端检测，以便客户可以使僵尸程序远离或降低僵尸程序的速度.

Akamai offers its customers a detection of web clients which are not browsers so that the customers can keep bots away or slowing bots down.

从>特殊的CURL问题可以看出网站SSL证书和抓取尝试导致403错误这种检测主要关心是否具有Accept-Language标头，具有值Keep-Alive的Connection标头和具有与Mozilla/...匹配的User-Agent.

As can be seen from Strange CURL issue with a particular website SSL certificate and Scraping attempts getting 403 error this kind of detection mainly cares about having a Accept-Language header, having a Connection header with the value Keep-Alive and having a User-Agent which matches Mozilla/....

这意味着以下代码更改将导致立即响应:

This means the following code changes result in an immediate response:

req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)
req.Header.Set("Connection","Keep-Alive")
req.Header.Set("Accept-Language","en-US")
req.Header.Set("User-Agent","Mozilla/5.0")

仍然，该网站显然不喜欢漫游器，您应该遵守这些愿望，并且不要过分强调该网站(例如进行大量信息收集).而且，由Akamai完成的僵尸程序检测可能会更改，恕不另行通知，即，即使此代码现在解决了问题，将来也可能不再起作用.如果许多客户端绕过了僵尸程序检测，则此类更改将尤其如此.

Still, the site obviously does not like bots and you should adhere to these wishes and not stress the site too much (like doing lots of information scraping). And, the bot detection done by Akamai might change without notice, i.e. even if this code fixes the problem now it might no longer work in the future. Such changes will be especially true if many clients bypass the bot detection.

这篇关于Golang Http获取请求非常慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！