多线程在C＃中有大量的Web请求 [英] Multithreading a large number of web requests in c#

问题描述

我有一个程序，我需要创建一些大量的文件夹到外部SharePoint网站（外部的意思我不能使用SharePoint对象模型）。 Web请求做工精良对于这一点，而只是做他们一次（发送请求，等待响应，重复）是相当缓慢的。我决定多线程请求，试图加快速度。该方案已经加快了很大，但一定量的时间后（1-2分钟左右之间），并发异常开始越来越抛出。

I have an program where I need to create some large number of folders to an external sharepoint site (external meaning I can't use the sharepoint object model). Web requests work well for this, but simply doing them one at a time (send request, wait for response, repeat) is rather slow. I decided to multithread the requests, to try and speed it up. The program has sped up considerably, but after some amount of time (between 1-2 minutes or so), concurrency exceptions start getting thrown.

code是低于，这是去最好的方法？

Code is below, is this the best way to go about this?

Semaphore Lock = new Semaphore(10, 10);
List<string> folderPathList = new List<string>();
//folderPathList populated

foreach (string folderPath in folderPathList)
{
    Lock.WaitOne();
    new Thread(delegate()
    {
        WebRequest request = WebRequest.Create(folderPath);
        request.Credentials = DefaultCredentials;
        request.Method = "MKCOL";

        WebResponse response = request.GetResponse();
        response.Close();
        Lock.Release();
    }).Start();
}
for(int i = 1;i <= 10;i++)
{
    Lock.WaitOne();
}

唯一的例外是沿

未处理的异常信息：System.Net.WebException：无法连接到远程服务器---> System.Net.Sockets.SocketException：每个套接字地址之一useage通常允许192.0.0.1:81结果
在System.Net.Sockets.Socket.DoConnect（端点endPointSnapshot，SocketAddre
SS的SocketAddress）结果
在System.Net.Sockets.Socket.InternalConnect（端点remoteEP）结果
在System.Net.ServicePoint.ConnectSocketInternal（布尔connectFailure，插座S4，S6插座，插座及放大器;插座，ip地址和放大器;地址，ConnectSocketState状态，
IAsyncResult的asyncResult，超时的Int32，异常和放大器;除外）

Unhandled Exception: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one useage of each socket address is normally permitted 192.0.0.1:81
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddre ss socketAddress)
at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

推荐答案

您可以创建太多的连接，从而利用了所有可以使用的本地端口。
有一个超时时间，当端口关闭后重新使用的。
的WebRequest 隐藏所有的低水平插座为你办理，但我猜测它最终耗尽的端口，或试图（重新）绑定到一个插座已经处于TIME_WAIT状态

You might create too many connections, thus using up all the local ports you can use. There's a timeout period for when a port can be reused after you close it. WebRequest hides all the low level socket handling for you, but I am guessing it eventually runs out of ports, or tries to (re)bind to a socket already in a TIME_WAIT state.

您应该确保你读了响应流，即使你不关心的响应。这应该有助于不产生过多的缠绵连接。

You should make sure you read the response stream, even if you don't care about the response. This should help not producing too many lingering connections.

WebResponse response = request.GetResponse();
new StreamReader(response.GetResponseStream()).ReadToEnd(); 

我会粘贴这里 一些相关信息：

I'll paste some relevant info from here:

当一个连接被关闭，在被关闭连接的5元组的一侧
{协议，本地IP，本地端口，远程IP，远程端口}进入240秒默认情况下，TIME_WAIT状态。
在这种情况下，该协议是固定的 - TCP
本地IP，远程IP和远程端口通常也固定。因此该变量是本地端口。
什么情况是，当你不绑定，在1024-5000的范围内的端口使用。
所以，大概有4000端口。如果您使用的所有的人都在4分 - 大致意思是你
每秒钟16 Web服务调用4分钟，你会用尽所有的端口。此异常的原因。

When a connection is closed, on the side that is closing the connection the 5 tuple { Protocol, Local IP, Local Port, Remote IP, Remote Port} goes into a TIME_WAIT state for 240 seconds by default. In this case, the protocol is fixed - TCP the local IP, remote IP and remote PORT are also typically fixed. So the variable is the local port. What happens is that when you don't bind, a port in the range 1024-5000 is used. So roughly you have 4000 ports. If you use all of them in 4 minutes - meaning roughly you make 16 web service calls per second for 4 minutes you will exhaust all the ports. That is the cause of this exception.

OK，现在这可怎么固定？

OK now how can this be fixed?

1. 的方法之一是增加了动态端口范围。默认情况下，最大为5000，您可以将此设为65534。
   HKLM \\系统\\ CurrentControlSet \\服务\\ TCPIP \\参数\\ MaxUserPort的是使用的关键。

1. One of the ways is to increase the dynamic port range. The max by default is 5000. You can set this up to 65534. HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort is the key to use.

您可以做的第二件事是，一旦连接并进入一个TIME_WAIT状态可以减少它是时间
在这种状态下，默认为4分钟，但你可以把这个30秒
HKLM \\系统\\ CurrentControlSet \\服务\\ TCPIP \\参数\\ TcpTimedWaitDelay的是使用的关键。
这个设置到30秒。

The second thing you can do is once the connection does get into an TIME_WAIT state you can reduce the time it is in that state, Default is 4 minutes, but you can set this to 30 seconds HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TCPTimedWaitDelay is the key to use. Set this to 30 seconds

这篇关于多线程在C＃中有大量的Web请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！