在keepAlive模式下了解HttpWebRequest [英] understand HttpWebRequest in keepAlive mode
问题描述
我知道这个话题已经讨论了很多次 但是我需要了解如何以正确的方式编写代码.
i know that the topic has been discussed many times but I need to understand how to write code in the correct way.
在协议版本为HTTP 1.1的情况下，我多次使用相同的HttpWebRequest(指向相同的url).
I use more times the same HttpWebRequest (to the same url) with protocol version HTTP 1.1.
Method ="POST"
Method = "POST"
KeepAlive = True
KeepAlive = True
但是每次我需要发送不同的请求并获得不同的响应时.
But everytime i need to send a different request, and get a different response.
(注意.下面的代码不正确，并引发异常)
(NB. This next code, it's not correct, and throw an exception)
Private Sub SendHttpWebReq()
Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)
httpWebReq.Method = "POST"
httpWebReq.KeepAlive = True
httpWebReq.ContentType = "application/x-www-form-urlencoded"
Dim myRequestString As New List(Of String) From {"abc", "def"}
Dim ContentList As New List(Of String)
For a = 0 To 1
Dim inputData As String = MyRequestString(a)
Dim postData As String = "firstone" + ChrW(61) + inputData
Dim encoding As New System.Text.ASCIIEncoding()
Dim byteData As Byte() = encoding.GetBytes(postData)
httpWebReq.ContentLength = byteData.Length
Dim newStream As IO.Stream = httpWebReq.GetRequestStream()
newStream.Write(byteData, 0, byteData.Length)
newStream.Flush()
newStream.Dispose()
Dim Response As Net.WebResponse = httpWebReq.GetResponse()
Dim ResponseStream As Io.Stream = Response.GetResponseStream()
Dim Content = New Io.MemoryStream()
ResponseStream.CopyTo(Content)
Response.Close()
Response.Dispose()
ResponseStream.Flush()
ResponseStream.Dispose()
ContentList.Add(System.Text.Encoding.UTF8.GetString(Content.ToArray))
Content = Nothing
Next
End Sub
当我运行代码时，第一次得到正确的响应，但是当我尝试重用" HttpWebRequest时，会抛出异常
When i run the code, the first time i get the correct response, but when i try to 'reuse' the HttpWebRequest, an Exception it's thrown
在线:
httpWebReq.ContentLength = byteData.Length
例外，即写入开始后无法设置此属性"
the exception it's 'This property cannot be set after writing has started'
搜索我发现了这个主题: 我可以重用HttpWebRequest吗?
searching i've found this topic: Am I able to reuse a HttpWebRequest?
在解释为重用" HttpWebRequest的地方，必须关闭流和WebResponse，然后我将其关闭，释放资源.
Where it's explained that for 'reuse' an HttpWebRequest, the stream and WebResponse must be closed, and i made it, releasing the resources.
Also in this topic it's explained the same thing: https://social.msdn.microsoft.com/Forums/en-US/8efad6b3-bc75-48f0-9858-8115dabb85c8/reusing-httpwebrequest-object?forum=netfxnetcom
But in this other topic: This property cannot be set after writing has started! on a C# WebRequest Object
一个成员说不可能重复使用HttpWebRequest.
a member says that it's not possible to reuse the HttpWebRequest.
我在重用"和创建一个新的"之间感到困惑.
I'm in confusion between 'reuse' and 'create a new one'.
我需要了解它指的是"KeepAlive":连接还是请求?
And i need to understand what 'KeepAlive' it's referred to: to the connection, or to the Request?
我想当我执行此指令时:
I suppose that when i execute this instruction:
Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)
我应该创建HttpWebRequest类的实例，
I should create an instance of HttpWebRequest class,
但是我应该使用此指令建立连接
But i should establish the connection with this instruction
Dim newStream As IO.Stream = httpWebReq.GetRequestStream()
我正确吗?
有帮助吗?
推荐答案
这是我认为需要澄清的内容，因为该声明可能被误导，其表达方式为:
This is what I think needs some clarifications, because this statement may be considered misleading, the way it's phrased:
1 WebRequest => 1 WebResponse.初始化后，您将无法再更改WebRequest中的任何内容.
1 WebRequest => 1 WebResponse. You can't change anything in a WebRequest once it has been initialized.
从原则上讲，这仍然有效，但是已初始化术语可能会造成混淆.更好的定义是:
This remains valid, in principle, but the initialized term could be confusing. A better definition, would be:
发出请求并返回WebResponse之后，直到关闭(处置)WebResponse后，您才能更改WebRequest的任何参数.
You can't change any parameter of a WebRequest after the request has been issued and a WebResponse has been returned, until after the WebResponse is closed (disposed).
WebResponse返回结果后，可以将其关闭-显式或隐式(在Using块中)-并且您可以请求另一个，并根据需要修改WebRequest参数(例如，将Method从POST到GET).
此外，必须在请求新的WebResponse之前重新初始化WebRequest .如果不这样做，它只会恢复为默认值.
After a WebResponse has returned a result, it can be closed - explicitly or implicitly (in a Using block) - and you can request another, modifying as necessary the WebRequest parameters (e.g. changing the Method from POST to GET).
More, a WebRequest must be re-initialized before requesting a new WebResponse. If you don't, it just falls back to it's defaults.
我在下面发布的代码是经典上下文(Web登录请求)的示例，其中WebRequest必须在同一过程中多次重新初始化，才能接收到不确定数量的WebRespons，直到目标地址为止(或到达网页).
The code I posted below, is an example of a classic context (a Web LogIn request) when a WebRequest must be re-initialized multiple times in the same procedure, to receive an undetermined number of WebResponses until a destination address (or landing page) is reached.
或多或少是以下架构:
This is, more or less, the schema:
--------------
(GET or POST) | WebRequest | (Method is POST)
|---------> | GET/(POST) | <-----------| <-------------- |
| -------------- | |
| | | |
-------------- --------------- ------------------ --------------
| New | | WebResponse |--> | LogIn Required |-->| LogIn |
| Location | --------------- ------------------ | Address |
| (Referer | | --------------
| Set) | |
-------------- (Set Cookies)
| |
| ---------------
| | LogIn |
Redirection <----| OK |---NO---|
--------------- |
| |
YES |
(Set Cookies) |
| Request
--------------- Denied
| Response | |
| URI | |
--------------- |
| |
EXIT <------------|
|
请注意，发出WebRequest时，如果访问请求的资源URI需要进行身份验证，则服务器可能不会通过StatusCode 302(已找到)，301(已移动)或303(已重定向)进行应答，它可能只是设置了StatusCode至200(确定).重定向是隐式的，因为设置了位置"标头，或者如果是WebForm登录名，则检索到的HTML页面将包含重定向.
Note that, issuing a WebRequest, if accessing the requested resource URI requires authentication, the server may NOT answer with a StatusCode 302 (Found), 301 (Moved) or 303 (Redirected), it might just set the StatusCode to 200 (OK). The redirection is implicit because the "Location" Header is set or, if it's a WebForm Login, the Html page retrieved contains the redirection.
无论如何，在检测到重定向之后，必须将新的重定向位置跟随到目的地.重定向可能包含一个或多个Hops，通常必须手动对其进行寻址(以验证是否将我们发送到了实际要去的地方).
Anyway, after a redirection has been detected, the new redirected location must be followed to destination. A redirection may consist of one or more Hops, which often must be addressed manually (to verify that we're sent where we actually want to go).
关于
keep-alive标头.客户端和/或服务器将
keep-alive标头设置为提示，对方应至少在一段时间内保持已建立的连接的打开状态，因为链接到当前事务的其他资源将被交换的可能性很高.这样可以防止创建大量昂贵的连接.
此设置在
Http和Ftp请求中很常见，并且是Http 1.1中的标准设置.About the
keep-alive Header.A
keep-alive header is set by the Client and/or the Server, to hint the counterpart that the established connection should be maintained open, at least for some time, because the chance that other resources, linked to the current transaction, will be exchanged is high.This prevents the creation of a possibly high number of costly connections.
This setup is common in
Http and Ftp requests and it's the standard in Http 1.1. 超文本传输协议(HTTP)保持活动标头(IETF)
与HTTP/1.0持久连接(IETF)的兼容性
Hypertext Transfer Protocol (HTTP) Keep-Alive Header (IETF)
Compatibility with HTTP/1.0 Persistent Connections (IETF)
需要记住的是，keep-alive标头是指使用WebRequest建立的连接，而不是WebRequest本身.创建连接以指定应保持打开状态时，后续请求应保持connection: keep-alive标头以符合协议.
在.NET HttpWebRequest中，将KeepAlive属性设置为False，等效于设置connection: close标头.
It needs to be remembered that the keep-alive header is referring to the Connection that has been established with a WebRequest, not to the WebRequest itself. When a connection is created specifing that it should be kept open, subsequent requests should maintain the connection: keep-alive Header to conform with the protocol.
In a .NET HttpWebRequest, setting the KeepAlive property to False, is equivalent to setting the connection: close Header.
但是，管理连接的逻辑和允许进程访问的连接池的逻辑是由 ServicePoint 作为每个连接请求的参考.
However, the logic which governs a Connection and the Connection-Pool which a process is granted access to, is managed by the ServicePointManager, using a ServicePoint as a reference for every connection request.
因此，WebRequest可以指定它要创建的Connection应该保持打开状态(因为它需要重复使用多次)，但是Connection背后的真正逻辑，即如何建立，维护和管理它，位于其他地方.
Thus, a WebRequest may specify that the Connection it is asking to create should be kept open (because it needs to be reused more times), but the real logic behind a Connection, how it is established, maintained and managed, lies somewhere else.
在Http 2.0中(StackOverflow/StackExchange使用此协议)，由于这个确切原因，完全忽略了keep-alive设置.连接逻辑是在更高的，独立的级别上进行管理的.
In Http 2.0 (StackOverflow/StackExchange use this protocol), the keep-alive setting is completely ignored, for this exact reason. A Connection logic is managed at a higher, independent, level.
(...)当您每3秒，每10秒调用1个新的HttpWebRequest时 秒还是每60秒?我寄那些东西有什么区别 请求是对还是错?
(...) when you call 1 new HttpWebRequest every 3 seconds, every 10 seconds, or every 60 seconds? What's the difference when i send those requests with True or False?
您将KeepAlive属性设置为提示，连接管理器应保持建立的连接处于打开状态，因为您知道该连接将被重用.但是，如果使用的协议提供了此设置，并且在控制连接池复杂性的内部逻辑所施加的限制内，ServicePointManager和远程服务器都将遵守该请求.
它应该在Http 1.0中设置，这是Http 1.1中的默认设置，在Http 2.0中将被忽略.
由于在建立连接之前您无法知道将使用哪种协议，因此通常将其设置为keep-alive，因为到请求资源的路由中的某些设备(特别是代理)可能需要此设置是明确的(已读).有关代理及其行为的IETF文件.
You set the KeepAlive property to hint the Connection Manager that the Connection established should be kept open, because you know it will be re-used. However, both the ServicePointManager and the remote Server will comply to the request if the protocol in use provides for this setting and within the limits imposed by the internal logic that governs the complex of the Connection-Pools.
It should be set in Http 1.0, it's the default setting in Http 1.1, it's ignored in Http 2.0.
Since you can't know which protocol will be used until a Connection is established, it's usually set to keep-alive, because some devices (Proxies, specifically) in the route to the requested resource, might need this setting to be explicit (read the IETF documents about Proxies and their behaviour).
在此示例中，
WebRequest在循环中重复初始化，并且每次Disposed都为Disposed，直到收到StatusCode 200(确定)或请求被拒绝或我们也被重定向很多次(取消令牌在这里也可能有用).In this example, a
WebRequest is repeatedly initialized in a Loop, and the underlying WebResponse is Disposed each time, until a StatusCode 200 (OK) is received or the request is denied or we have been redirected too many times (a Cancellation Token may also be useful here).在该示例中，应以这种方式调用main方法:
In the example, the main method is meant to be called this way:
Public Async Sub SomeMethodAsync()
LoginParameters = New LoginObject() With {
.CookieJar = New CookieContainer,
.LogInUrl = "[Some IP Address]",
.Credentials = New Dictionary(Of String, String)
}
LoginParameters.Credentials.Add("UserName", "[Username]")
LoginParameters.Credentials.Add("Email", "[email]")
LoginParameters.Credentials.Add("Password", "[Password]")
LoginParameters = Await HttpLogIn(LoginParameters)
End Sub
LoginParameters对象必须保留，因为引用了CookieContainer，其中包含在身份验证之后收到的Cookies.初始化新的WebRequest时，会将这些Cookie传递到服务器，以证明"请求的凭据已通过身份验证.请注意，这些Cookie会在一段时间后过期(除非发出会话限制，否则在发出新的WebRequest时会刷新").在这种情况下，将自动重复登录过程.
The LoginParameters object must be preserved, because references a CookieContainer, which contains the Cookies received after the authentication. These Cookies are passed to the server when a new WebRequest is initialized, as a "proof" that the request's credentials are already authenticated. Note that these Cookies expire after a while (thay are "refreshed" when a new WebRequest is issued, unless the Session has a time limit). If this is the case, the Login procedure is automatically repeated.
Imports System.Net
Imports System.Net.Security
Imports System.IO
Imports System.Security
Imports System.Security.Cryptography
Imports System.Security.Cryptography.X509Certificates
Imports System.Text
Public LoginParameters As LoginObject
Public Class LoginObject
Public Property LogInUrl As String
Public Property ResponseUrl As String
Public Property Credentials As Dictionary(Of String, String)
Public Property StatusCode As HttpStatusCode
Public Property CookieJar As New CookieContainer()
End Class
Public Async Function HttpLogIn(LogInParameters As LoginObject) As Task(Of LoginObject)
Dim httpRequest As HttpWebRequest
Dim StatusCode As HttpStatusCode
Dim MaxHops As Integer = 20
' Windows 7 (.Net 4.5.1+ required):
'ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
' Windows 10 (.Net 4.5.1+ required):
ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault
'If needed or for testing
'ServicePointManager.ServerCertificateValidationCallback = AddressOf CertificateValidation
httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)
Try
HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)
Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
StatusCode = httpResponse.StatusCode
End Using
If StatusCode = HttpStatusCode.OK OrElse StatusCode = HttpStatusCode.NoContent Then
'POST Parameters are URLEncoded and the encoded strings converted to a Byte array of UTF8 chars
Dim EncodedParameters As Byte() = HTTP_EncodePOSTParameters(LogInParameters.Credentials)
httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)
httpRequest.Method = WebRequestMethods.Http.Post
httpRequest.ContentType = "application/x-www-form-urlencoded"
httpRequest.ContentLength = EncodedParameters.Length
HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)
Using stream As Stream = Await httpRequest.GetRequestStreamAsync()
stream.Write(EncodedParameters, 0, EncodedParameters.Length)
End Using
Dim Hops As Integer = 0
Dim Referer As String = LogInParameters.LogInUrl
Dim LastHttpMethod As String = httpRequest.Method
Do
'Evaluate Authentication redirect or page moved
Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
StatusCode = httpResponse.StatusCode
LogInParameters.ResponseUrl = URIFromResponseLocation(httpResponse).ToString()
End Using
If (StatusCode = HttpStatusCode.Moved) OrElse
(StatusCode = HttpStatusCode.Found) OrElse
(StatusCode = HttpStatusCode.RedirectMethod) OrElse
(StatusCode = HttpStatusCode.RedirectKeepVerb) Then
httpRequest = WebRequest.CreateHttp(LogInParameters.ResponseUrl)
HTTP_RequestHeadersInit(httpRequest, Referer, LogInParameters.CookieJar)
If StatusCode = HttpStatusCode.RedirectKeepVerb Then
httpRequest.Method = LastHttpMethod
Else
LastHttpMethod = httpRequest.Method
End If
End If
If (CType(StatusCode, Integer) > 320) OrElse Hops >= MaxHops Then
Exit Do
End If
Hops += 1
Loop While (StatusCode <> HttpStatusCode.OK)
If StatusCode = HttpStatusCode.OK Then
LogInParameters.CookieJar = httpRequest.CookieContainer
End If
End If
Catch exW As WebException
StatusCode = If(exW.Response IsNot Nothing,
CType(exW.Response, HttpWebResponse).StatusCode,
CType(exW.Status, HttpStatusCode))
Catch exS As System.Exception
StatusCode = CType(WebExceptionStatus.RequestCanceled, HttpStatusCode)
Finally
ServicePointManager.ServerCertificateValidationCallback = Nothing
End Try
LogInParameters.StatusCode = StatusCode
Return LogInParameters
End Function
Private Sub HTTP_RequestHeadersInit(ByRef httpReq As HttpWebRequest,
Referer As String,
CookiesJar As CookieContainer)
httpReq.Date = DateTime.Now
httpReq.CookieContainer = CookiesJar
httpReq.KeepAlive = True
httpReq.ConnectionGroupName = Guid.NewGuid().ToString()
httpReq.AllowAutoRedirect = False
httpReq.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
httpReq.ServicePoint.Expect100Continue = False
httpReq.Referer = Referer
httpReq.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"
httpReq.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
httpReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.9,en;q=0.5")
httpReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
httpReq.Headers.Add(HttpRequestHeader.CacheControl, "no-cache")
End Sub
Private Function HTTP_EncodePOSTParameters(PostParameters As Dictionary(Of String, String)) As Byte()
Dim Encoder As New System.Text.UTF8Encoding()
Dim CredentialValues As New StringBuilder()
Dim _first As Boolean = True
For Each CurrentKeyPair As KeyValuePair(Of String, String) In PostParameters
If _first = False Then CredentialValues.Append("&")
CredentialValues.AppendFormat("{0}={1}", WebUtility.UrlEncode(CurrentKeyPair.Key),
WebUtility.UrlEncode(CurrentKeyPair.Value))
_first = False
Next
Return Encoder.GetBytes(CredentialValues.ToString())
End Function
Private Function URIFromResponseLocation(Response As HttpWebResponse) As System.Uri
Dim uri As Uri
Dim Location As String = Response.Headers("Location")
Try
If uri.IsWellFormedUriString(Location, UriKind.Absolute) Then
uri = New Uri(Location, UriKind.Absolute)
Else
Dim HostUri As String = Response.ResponseUri.GetComponents(UriComponents.SchemeAndServer,
UriFormat.Unescaped) + Location
uri = If(uri.IsWellFormedUriString(HostUri, UriKind.Absolute),
New Uri(HostUri),
New Uri(Response.ResponseUri.GetComponents(UriComponents.Scheme, UriFormat.Unescaped) +
Response.ResponseUri.Host + Location))
End If
Catch ExceptionOnInvalidUri As Exception
uri = New Uri(Location, UriKind.Relative)
End Try
Return uri
End Function
Private Function CertificateValidation(sender As Object,
CACert As X509Certificate,
CAChain As X509Chain,
PolicyErrors As SslPolicyErrors) As Boolean
'This method, as it is, accepts a Server certificate in any case
'It could be eventually adapted to refuse a connection (returning false)
'if the certificate is invalid, expired or from a untrusted path
If (PolicyErrors = SslPolicyErrors.None) Then Return True
'If a Certificated must be added to the Chain, uncomment the code below,
'selecting a Certificate in the Local (or other) Storage
'Dim MyCert As X509Certificate2 = New X509Certificate2("[localstorage]/[ca.cert]")
'CAChain.ChainPolicy.ExtraStore.Add(MyCert)
'CAChain.Build(MyCert)
'For Each CACStatus As X509ChainStatus In CAChain.ChainStatus
' If (CACStatus.Status <> X509ChainStatusFlags.NoError) And
' (CACStatus.Status <> X509ChainStatusFlags.UntrustedRoot) Then
' Return False
' End If
'Next
Return True
End Function
这篇关于在keepAlive模式下了解HttpWebRequest的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
