快速高效地下载多个文件(异步) [英] Downloading multiple files by fastly and efficiently(async)

问题描述

我有很多文件需要下载.因此，我尝试使用如下所示的新异步功能.

I have so many files that i have to download. So i try to use power of new async features as below.

var streamTasks = urls.Select(async url => (await WebRequest.CreateHttp(url).GetResponseAsync()).GetResponseStream()).ToList();

var streams = await Task.WhenAll(streamTasks);
foreach (var stream in streams)
{
    using (var fileStream = new FileStream("blabla", FileMode.Create))
    {
        await stream.CopyToAsync(fileStream);
    }
}

我担心此代码会导致大内存使用，因为如果有1000个文件包含2MB文件，那么此代码会将1000 * 2MB流加载到内存中?

What i am afraid of about this code it will cause big memory usage because if there are 1000 files that contains 2MB file so this code will load 1000*2MB streams into memory?

我可能会错过一些东西，或者我完全正确.如果不遗漏任何东西，那么最好等待每个请求并消耗流是最好的方法?

I may missing something or i am totally right. If am not missed something so it is better to await every request and consume stream is best approach ?

推荐答案

两个选项都可能有问题.一次只下载一个文件不会扩展，并且会花费一些时间，而一次下载所有文件可能会带来很大的负担(此外，无需等待所有文件下载后再进行处理).

Both options could be problematic. Downloading only one at a time doesn't scale and takes time while downloading all files at once could be too much of a load (also, no need to wait for all to download before you process them).

我更喜欢始终以可配置的大小来限制此类操作.一种简单的方法是使用 AsyncLock (使用 TPL Dataflow 与 MaxDegreeOfParallelism .

I prefer to always cap such operation with a configurable size. A simple way to do so is to use an AsyncLock (which utilizes SemaphoreSlim). A more robust way is to use TPL Dataflow with a MaxDegreeOfParallelism.

var block = new ActionBlock<string>(url =>
    {
        var stream = (await WebRequest.CreateHttp(url).GetResponseAsync()).GetResponseStream();
        using (var fileStream = new FileStream("blabla", FileMode.Create))
        {
            await stream.CopyToAsync(fileStream);
        }
    },
    new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 100 });

这篇关于快速高效地下载多个文件(异步)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！