用C#快速下载Html源代码 [英] Quick Download HTML Source in C#

问题描述

我正在尝试用C#从单个网站(https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription/)下载一个HTML源代码。

问题是下载一个30kb的HTML页面源代码需要10秒钟。互联网连接不是问题，因为我可以在此程序中立即下载10MB文件。

已在单独的线程和主线程中执行了以下代码。下载仍然需要10-12秒。

---

1)

using (var httpClient = new HttpClient())
    {
        using (var request = new HttpRequestMessage(new HttpMethod("GET"), url))
        {
            var response = await httpClient.SendAsync(request);
        }
    }

---

2)

using (var client = new System.Net.WebClient())
    {
        client.Proxy = null;
        response = client.DownloadString(url);
    }

---

3)

using (var client = new System.Net.WebClient())
    {
        webClient.Proxy = GlobalProxySelection.GetEmptyWebProxy();
        response = client.DownloadString(url);
    }

---

4)

WebRequest.DefaultWebProxy = null;

using (var client = new System.Net.WebClient())
    {
        response = client.DownloadString(url);
    }

---

5)

var client = new WebClient()
response = client.DownloadString(url);

---

6)

var client = new WebClient()
client.DownloadFile(url, filepath);

---

7)

System.Net.WebClient myWebClient = new System.Net.WebClient();
WebProxy myProxy = new WebProxy();
myProxy.IsBypassed(new Uri(url));
myWebClient.Proxy = myProxy;
response = myWebClient.DownloadString(url);

---

8)

using var client = new HttpClient();
var content = await client.GetStringAsync(url);

---

9)

HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(Url);
myRequest.Method = "GET";
WebResponse myResponse = myRequest.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream(), System.Text.Encoding.UTF8);
string result = sr.ReadToEnd();
sr.Close();
myResponse.Close();

---

我希望在C#中使用更快的方法来完成此操作。

我们非常感谢您能提供的任何信息或帮助。

推荐答案

我知道这是过时的，但我想我找到了原因：我在其他站点遇到过这种情况。如果您查看响应cookie，您会发现一个名为ak_bmsc的cookie。该Cookie显示该站点正在运行Akamai Bot管理器。它提供僵尸程序保护，从而阻止"看起来"可疑的请求。

为了从主机获得快速响应，您需要正确的请求设置。在本例中：

• 标题：
  • Host：(其主机数据)www.faa.gov
  • Accept：(类似：)*/*
• 曲奇：
  • AkamaiEdge = true

示例：

class Program
    {
        private static readonly HttpClient _client = new HttpClient();
        private static readonly string _url = "https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription/";

        static async Task Main(string[] args)
        {
            var sw = Stopwatch.StartNew();
            using (var request = new HttpRequestMessage(HttpMethod.Get,_url))
            {
                request.Headers.Add("Host", "www.faa.gov");
                request.Headers.Add("Accept", "*/*");
                request.Headers.Add("Cookie", "AkamaiEdge=true");
                Console.WriteLine(await _client.SendAsync(request));
            }
            Console.WriteLine("Elapsed: {0} ms", sw.ElapsedMilliseconds);
        }
    }

我需要896毫秒。

顺便说一句，您不应该将HttpClient放在Using块中。我知道它是一次性的，但它不是为丢弃而设计的。

这篇关于用C#快速下载Html源代码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！