在HttpWebRequest的并发限制 [英] Concurrency Limit on HttpWebRequest
问题描述
我写来衡量我的速度有多快可以下载使用C＃网页的应用程序。我公司供应的唯一域名列表，然后我生成线程的X个，并执行HTTPWebRequests直到域列表已经被消耗掉。问题是，不管我有多少线程使用，我只得到约每秒3页。
我发现System.Net.ServicePointManager.DefaultConnectionLimit为2，但我的印象是，这是关系到每个域的连接数。因为列表中的每个域是独一无二的，这不应该是一个问题。
然后我发现的GetResponse（）方法阻止所有其他进程访问，直到WebResponse的关闭：的http://www.codeproject.com/KB/IP/Crawler.aspx#WebRequest ，我还没有在网上找到了支持这个要求的任何其他信息。起来，但我实现了使用套接字的HTTP请求，我注意到一个显著加快（4倍至6倍）
所以我的问题：没有人知道究竟是如何的HttpWebRequest对象的工作？有没有除了什么上面提到的一个解决办法？还是有C＃编写的地方？
你有没有如BeginGetResponse尝试使用异步方法（）？
如果您正在使用.NET 4.0，你可以试试这个代码。基本上我使用任务，以对特定站点1000个请求（我用这个做应用程序的负载测试我的开发机上，我看到没有限制这样，因为我的应用程序是看到快速连续这些请求）
公共部分Form1类：表格
{
公共Form1中（）
{
的InitializeComponent（）;
}
私人无效的button1_Click（对象发件人，EventArgs五）
{
的for（int i = 0; I< 1000;我++）
{
变种的WebRequest = WebRequest.Create（textBox1.Text）;
webRequest.GetReponseAsync（）ContinueWith（T =>
{
如果（t.Exception ==使用空）
{
（VAR SR =新的StreamReader （t.Result.GetResponseStream（）））
{
字符串str = sr.ReadToEnd（）;
}
}
，否则
System.Diagnostics程序.Debug.WriteLine（t.Exception.InnerException.Message）;
}）;
}
}
}
公共静态类WebRequestExtensions
{
公共静态任务<&WebResponse的GT; GetReponseAsync（此要求的WebRequest）
{
返回Task.Factory.FromAsync<&WebResponse的GT;（request.BeginGetResponse，request.EndGetResponse，NULL）;
}
}
由于这里的工作量为I / O密集型，产卵线程完成这项工作不是必需的，实际上可能会损害性能。上使用WebClient类使用I / O完成端口的异步方法，因此会更高性能和耗资源少。
I am writing an application to measure how fast I can download web pages using C#. I supply a list of unique domain names, then I spawn X number of threads and perform HTTPWebRequests until the list of domains has been consumed. The problem is that no matter how many threads I use, I only get about 3 pages per second.
I discovered that the System.Net.ServicePointManager.DefaultConnectionLimit is 2, but I was under the impression that this is related to the number of connections per domain. Since each domain in the list is unique, this should not be an issue.
Then I found that the GetResponse() method blocks access from all other processes until the WebResponse is closed: http://www.codeproject.com/KB/IP/Crawler.aspx#WebRequest, I have not found any other information on the web to back this claim up, however I implemented a HTTP request using sockets, and I noticed a significant speed up (4x to 6x).
So my questions: does anyone know exactly how the HttpWebRequest objects work?, is there a workaround besides what was mentioned above?, or are there any examples of high speed web crawlers written in C# anywhere?
Have you tried using the async methods such as BeginGetResponse() ?
If you're using .net 4.0 you may want to try this code. Essentially I use Tasks to make 1000 requests on a specific site (I use this to do load testing of app on my dev machine and I see no limits as such since my app is seeing these requests in rapid succession)
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
for (int i = 0; i < 1000; i++)
{
var webRequest = WebRequest.Create(textBox1.Text);
webRequest.GetReponseAsync().ContinueWith(t =>
{
if (t.Exception == null)
{
using (var sr = new StreamReader(t.Result.GetResponseStream()))
{
string str = sr.ReadToEnd();
}
}
else
System.Diagnostics.Debug.WriteLine(t.Exception.InnerException.Message);
});
}
}
}
public static class WebRequestExtensions
{
public static Task<WebResponse> GetReponseAsync(this WebRequest request)
{
return Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null);
}
}
Since the workload here is I/O bound, spawning threads to get the job done is not required and in fact could hurt performance. Using the Async methods on the WebClient class use I/O completion ports and so will be much more performant and less resource hungry.
这篇关于在HttpWebRequest的并发限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
