C# - 如何同时执行多个 Web 请求 [英] C# - how to do multiple web requests at the same time

问题描述

我写了一个代码来检查 url，但是，ir 的工作真的很慢./p>

我的代码:

Parallel.ForEach(urls, new ParallelOptions {MaxDegreeOfParallelism = 10}, s =>{尝试 {使用(HttpRequest httpRequest = new HttpRequest()) {httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0";httpRequest.Cookies = 新的 CookieDictionary(false);httpRequest.ConnectTimeout = 10000;httpRequest.ReadWriteTimeout = 10000;httpRequest.KeepAlive = true;httpRequest.IgnoreProtocolErrors = true;字符串检查 = httpRequest.Get(s + "'", null).ToString();if (errors.Any(new Func < string, bool > (check.Contains))) {Valid.Add(s);Console.WriteLine(s);File.WriteAllLines(Environment.CurrentDirectory + "/Good.txt", Valid);}}} 抓住 {}});

解决方案

您的服务调用不太可能是 CPU 绑定.因此，旋转更多线程来处理负载可能不是最好的方法——如果您使用 async 和 await 代替，您将获得更好的吞吐量，如果可以，使用更现代的 HttpClient 而不是 HttpRequest 或 HttpWebRequest.

这是一个如何做的例子:

var client = new HttpClient();//从一个URL列表开始var urls = 新字符串[]{http://www.google.com"，http://www.bing.com"};//开始对所有的请求var 请求 = urls.Select(网址 =>client.GetAsync(url)).ToList();//等待所有请求完成等待 Task.WhenAll(requests);//获取响应var 响应 = requests.Select(任务 =>任务.结果);foreach(响应中的var r){//提取消息体var s = 等待 r.Content.ReadAsStringAsync();Console.WriteLine(s);}

I wrote a code to check urls, however, ir works really slow.. I want to try to make it work on few urls at the same time, for example 10 urls or at least make it as fast as possible.

my Code:

Parallel.ForEach(urls, new ParallelOptions {
  MaxDegreeOfParallelism = 10
}, s => {
  try {
    using(HttpRequest httpRequest = new HttpRequest()) {
      httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0";
      httpRequest.Cookies = new CookieDictionary(false);
      httpRequest.ConnectTimeout = 10000;
      httpRequest.ReadWriteTimeout = 10000;
      httpRequest.KeepAlive = true;
      httpRequest.IgnoreProtocolErrors = true;
      string check = httpRequest.Get(s + "'", null).ToString();
      if (errors.Any(new Func < string, bool > (check.Contains))) {
        Valid.Add(s);
        Console.WriteLine(s);
        File.WriteAllLines(Environment.CurrentDirectory + "/Good.txt", Valid);
      }
    }
  } catch {

  }
});

解决方案

It is unlikely that your service calls are CPU-bound. So spinning up more threads to handle the load is maybe not the best approach-- you will get better throughput if you use async and await instead, if you can, using the more modern HttpClient instead of HttpRequest or HttpWebRequest.

Here is an example of how to do it:

var client = new HttpClient();

//Start with a list of URLs
var urls = new string[]
    {
        "http://www.google.com",
        "http://www.bing.com"
    };

//Start requests for all of them
var requests  = urls.Select
    (
        url => client.GetAsync(url)
    ).ToList();

//Wait for all the requests to finish
await Task.WhenAll(requests);

//Get the responses
var responses = requests.Select
    (
        task => task.Result
    );

foreach (var r in responses)
{
    // Extract the message body
    var s = await r.Content.ReadAsStringAsync();
    Console.WriteLine(s);
}

这篇关于C# - 如何同时执行多个 Web 请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！