在Go中没有重定向的查询网址 [英] Query URL without redirect in Go

问题描述

我为重定向脚本编写基准测试。

我使用我的程序来查询某个重定向到AppStore的URL。但我不想下载AppStore页面。我只希望记录重定向网址或错误。

如何在不使用第二次重定向查询的情况下转到查询网址？

$ b 更新

我尝试了两种解决方案。我正在做基准测试。
我运行1个或多个去程序，使用10 - 500个程序。他们在一个循环中查询URL。
我的服务器也是用go编写的。它会报告每秒的请求数。

• 第一个解决方案： http.DefaultTransport.RoundTrip - 工作缓慢，给出错误。
  前4秒工作正常。进行300-500个查询，性能下降到每秒80个查询。

然后下降到每秒0-5个查询，并且查询脚本开始得到像

的错误

  dial tcp IP：80：连接尝试失败，因为连接的
方在一段时间后没有正确响应，或者建立的
连接失败，因为连接的主机无法响应。 
  

我想它会重新使用已关闭的连接。

• 第二种解决方案： CheckRedirect字段可以在不断的表现下工作。我不确定它是否重新使用连接，或者是否为每个请求打开一个新的连接。我为循环中的每个请求创建 client 。它是如何在现实生活中表现出来的（每一个请求都是一个新的连接）。有没有办法确保连接在每个查询后关闭并且不被重用？

这就是为什么我要标记第二个解决方案，以回答我的问题。但对于我的研究来说，每个查询都是一个新连接是非常重要的。如何确保使用第二种解决方案？

解决方案

为了完整起见，您可以 code> http.Client ，不遵循重定向。 http.Client 有一个 CheckRedirect code>字段这是一个函数。



如果这个函数返回一个错误，那么 httpClient.Do（...） will not 遵循重定向（请参阅 go的源代码中的doFollowingRedirects（）函数），而是会返回一个错误（它的具体类型将是 url.Error ，其 URL 字段将是重定向到URL，也就是位置标题值，查看此代码）。

您可以看到我的 gocrawl library ，以获取此用法的具体示例。

I am writing a benchmark test for a redirect script.

I wisg my program to query certain URL that redirects to AppStore. But I do not wish to download AppStore page. I just wish to log redirect URL or error.

How do I tell Go to query URL without second redirect query?

---

UPDATE

Both answers are correct BUT:

I tried both solutions. I am doing benchmarking. I run 1 or many go processes with 10 - 500 go routines. They query URL in a loop. My server is also written in go. It reports number of requests every second.

• First solution: http.DefaultTransport.RoundTrip - works slow, gives errors. First 4 seconds works fine. Making 300-500 queries then performance drops to 80 query per second.

Then drops to 0-5 query per second and queryies script start getting errors like

dial tcp IP:80: A connection attempt failed because the connected 
party did not properly respond after a period of time, or established 
connection failed because connected host has failed to respond.

I guess it re-use connection that is closed.

• Second solution: CheckRedirect field works with constant performance. I am not sure if it re-uses connections or it opens a new connection for every request. I create client for every request in a loop. It is how it will behave in a real life (every request is a new connection). Is there way to ensure that connections are closed after each query and not re-used?

That is why I am going to mark second solution as such that answer my question. But for my research it is very important that each query was a new connection. How can I ensure with second solution?

解决方案

For completeness' sake, you can use an http.Client and not follow redirects. http.Client has a CheckRedirect field which is a function. It is called before following any redirection.

If this function returns an error, then httpClient.Do(...) will not follow the redirect (see doFollowingRedirects() function in Go's source code) and instead will return an error (its concrete type will be url.Error, and its URL field will be the redirect-to URL, aka the Location header value, see this code).

You can see my gocrawl library for a concrete example of this use.

这篇关于在Go中没有重定向的查询网址的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！