HttpClient TPL并行多请求最快方法 [英] Httpclient tpl parallel multiple request fastest way

问题描述

我要下载URL列表的WebPages内容(10000个URL).

I want to download webPages content of url list (10 000 urls).

1. httpCLient是最快，最干净的方法(而不是httpwebrequest或webclient)吗?
2. 如果我想快点，TPL是最好的方法吗?

我正在寻找类似的东西，但确实又快又干净(万次请求)?

I'm looking for something like, but really fast and clean (10 000 request) ?

public List<string> GetContentListOfUrlList(List<Uri> uriList, int maxSimultaneousRequest)
    {
        //requesting url by the fastest way

    }

我希望这样会更好；)

根据noseratio 其他帖子 是最好的解决方案吗?

EDIT 2 : According to noseratio other post Is the best solution ?

public async Task<List<string>> DownloadAsync(List<Uri> urls, int maxDownloads)
    {
        var concurrentQueue = new ConcurrentQueue<string>();

        using (var semaphore = new SemaphoreSlim(maxDownloads))
        using (var httpClient = new HttpClient())
        {
            var tasks = urls.Select(async (url) =>
            {
                await semaphore.WaitAsync();
                try
                {
                    var data = await httpClient.GetStringAsync(url);
                    concurrentQueue.Enqueue(data);
                }
                finally
                {
                    semaphore.Release();
                }
            });

            await Task.WhenAll(tasks.ToArray());
        }
        return concurrentQueue.ToList();
    }

问题

1. configureAwait?我应该使用

1. configureAwait? Should I use

var data = await httpClient.GetStringAsync(url).ConfigureAwait(false);

var data = await httpClient.GetStringAsync(url).ConfigureAwait(false);

var data =等待httpClient.GetStringAsync(url);

var data = await httpClient.GetStringAsync(url);

1. ServicePointManager.DefaultConnectionLimit?我也应该更改此属性吗?

推荐答案

有一个 ParallelOptions.MaxDegreeOfParallelism 属性，用于指定并发操作的最大数量:

There is a ParallelOptions.MaxDegreeOfParallelism Property which specifies the maximum number of concurrent operations:

Parallel.ForEach(list, 
        new ParallelOptions { MaxDegreeOfParallism = 4 }, 
        DownloadPage);

参考: 查看全文