异步和并行下载文件 [英] Asynchronously and parallelly downloading files

查看：119
本文介绍了异步和并行下载文件的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

修改

我已经改变了问题的标题，以反映的问题我有，但是还对如何方便地实现这个答案。

我试图让第二个方法返回任务＆LT; TResult＆GT; 而不是任务作为在第1种方法但我得到错误的级联作为试图修复它的结果。


  • 我加入收益在等待体（partition.Current）;

  • 在反过来要求我先下添加​​，所以我增加了一个return语句返回NULL 下面

  • 但是，现在的select语句抱怨说，它不能推断从查询
    • 的类型参数
  • 更改 Task.Run 到 Task.Run＆LT; TResult方式＆gt; 但都没有成功

我怎样才能解决这个问题？

第一种方法来自<一个href=\"http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx\">http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx,第二种方法是，我试图创造过载。

 公共静态类扩展
{
    公共静态任务ForEachAsync＆LT; T＆GT;（这个IEnumerable的＆LT; T＆GT;源，INT DOP，Func键＆LT; T，任务＆GT;体）
    {
        返回Task.WhenAll（
            从Partitioner.Create（源）.GetPartitions（DOP）分区
            选择Task.Run（异步委托
            {
                使用（分区）
                    而（partition.MoveNext（））
                        等待体（partition.Current）;
            }））;
    }    公共静态任务ForEachAsync＆LT; T，TResult＆GT;（这个IEnumerable的＆LT; T＆GT;源，INT DOP，Func键＆LT; T，任务＆LT; TResult＆GT;＆GT;体）
    {
        返回Task.WhenAll（
            从Partitioner.Create（源）.GetPartitions（DOP）分区
            选择Task.Run（异步委托
            {
                使用（分区）
                    而（partition.MoveNext（））
                        等待体（partition.Current）;
            }））;
    }
}

用法示例：

通过这种方法，我想下载多个文件并行异步：

 专用异步无效MainWindow_Loaded（对象发件人，RoutedEventArgs E）
{
    艺术家艺术家=等待GetArtist（）;
    IEnumerable的＆LT;串GT;枚举= artist.Reviews.Select（S =＆GT; s.ImageUrl）;
    字符串[] downloadFile =等待DownloadFiles（枚举）;
}公共静态异步任务＆LT;字符串[]＆GT; DownloadFiles（IEnumerable的＆LT;串GT;枚举）
{
    如果（枚举== NULL）抛出新的ArgumentNullException（枚举）;
    等待enumerable.ForEachAsync（5，S =＆GT; DownloadFile（S））;
    //不完整，上面的语句是无效的，不能退换
}公共静态异步任务＆LT;串GT; DownloadFile（字符串地址）
{
    / *从指定的地址下载一个文件，
        *上失败是成功还是返回null目标文件名* /    如果（地址== NULL）
    {
        返回null;
    }    乌里结果;
    如果（！Uri.TryCreate（地址，UriKind.Absolute，出结果））
    {
        的Debug.WriteLine（的String.Format（无法​​从指定的地址创建URI：{0}，地址））;
        返回null;
    }    尝试
    {
        使用（VAR的客户=新的WebClient（））
        {
            字符串文件名= Path.GetTempFileName（）;
            等待client.DownloadFileTaskAsync（地址，文件名）;
            的Debug.WriteLine（的String.Format（下载保存到文件：{0}（{1}），文件名，地址））;
            返回文件名;
        }
    }
    赶上（引发WebException引发WebException）
    {
        的Debug.WriteLine（的String.Format（无法​​从指定的地址下载文件：{0}，webException.Message））;
        返回null;
    }
}


解决方案

我解决了它，并在这里张贴，可以帮助有同样的问题的人。

我最初的需求是一个小帮手，将快速地下载图像，而且只是下降的连接​​，如果服务器没有迅速作出反应，这一切的并行异步

这帮手将返回包含远程路径，本地路径，如果发生了一个异常的元组;所以它总是好的，知道为什么错误的下载量已经出现故障非常有用。我想我忘了没有任何可能发生的下载情况，但欢迎您发表评论吧。


  • 您指定的URL列表下载

  • 您可以指定将被保存在本地的文件名，如果没有人会为你生成

  • 可选取消下载的持续时间（为方便缓慢或无法访问服务器）

您可以只用 DownloadFileTaskAsync 本身或使用 ForEachAsync 帮手并行和异步下载。

code对如何使用它的例子：

 专用异步无效MainWindow_Loaded（对象发件人，RoutedEventArgs E）
{
    IEnumerable的＆LT;串GT;枚举=您的网址在这里;
    VAR的结果=新的List＆LT元组LT;字符串，字符串，异常＆GT;＆GT;（）;
    等待enumerable.ForEachAsync（S = GT; DownloadFileTaskAsync（S，空，1000），（URL，T）=＆GT; results.Add（T））;
}///＆LT;总结＆gt;
///下载总数从指定的因特网地址的文件。
///＆LT; /总结＆gt;
///＆所述; PARAM NAME =remotePath＆gt;本文件的因特网地址下载＆所述; /参数＆GT;
///＆LT; PARAM NAME =的localPath＆GT;
///本地的文件名，其中存储下载的内容，如果为null一个临时文件名会
///产生。
///＆LT; /参数＆GT;
///＆LT; PARAM NAME =TIMEOUT方式＆gt;持续时间以毫秒为单位取消操作之前和LT; /参数＆GT;
///＆LT;收益方式＆gt; /回报＆gt;将包含远程路径，本地路径和如果发生异常＆LT元组;
私有静态异步任务＆LT元组LT;字符串，字符串，异常＆GT;＆GT; DownloadFileTaskAsync（字符串remotePath，
    字符串的localPath = NULL，INT超时= 3000）
{
    尝试
    {
        如果（remotePath == NULL）
        {
            的Debug.WriteLine（DownloadFileTaskAsync（空远程路径）：跳过）;
            抛出新的ArgumentNullException（remotePath）;
        }        如果（的localPath == NULL）
        {
            的Debug.WriteLine（
                的String.format（
                    DownloadFileTaskAsync（空本地路径）：用于产生一个临时文件名{0}，
                    remotePath））;
            的localPath = Path.GetTempFileName（）;
        }        使用（VAR的客户=新的WebClient（））
        {
            TimerCallback timerCallback = C =＆GT;
            {
                VAR的WebClient =（Web客户端）C;
                如果回报（webClient.IsBusy！）;
                webClient.CancelAsync（）;
                的Debug.WriteLine（的String.Format（DownloadFileTaskAsync（由于超时）：{0}，remotePath））;
            };
            使用（VAR定时器=新定时器（timerCallback，客户端，超时Timeout.Infinite））
            {
                等待client.DownloadFileTaskAsync（remotePath，localPath来）;
            }
            的Debug.WriteLine（的String.Format（DownloadFileTaskAsync（下载）：{0}，remotePath））;
            返回新行＆LT;字符串，字符串，异常＆GT;（remotePath，localPath来，NULL）;
        }
    }
    赶上（异常前）
    {
        返回新行＆LT;字符串，字符串，异常＆GT;（remotePath，空，前）;
    }
}公共静态类扩展
{
    公共静态任务ForEachAsync＆LT; TSource，TResult＆GT;（
        这IEnumerable的＆LT; TSource＆GT;资源，
        FUNC＆LT; TSource，任务＆LT; TResult＆GT;＆GT; taskSelector，动作＆LT; TSource，TResult＆GT; resultProcessor）
    {
        变种oneAtATime =新SemaphoreSlim（5，10）;
        返回Task.WhenAll（
            从源项
            选择ProcessAsync（项目，taskSelector，resultProcessor，oneAtATime））;
    }    私有静态异步任务ProcessAsync＆LT; TSource，TResult＆GT;（
        TSource项目，
        FUNC＆LT; TSource，任务＆LT; TResult＆GT;＆GT; taskSelector，动作＆LT; TSource，TResult＆GT; resultProcessor，
        SemaphoreSlim oneAtATime）
    {
        TResult结果=等待taskSelector（项目）;
        等待oneAtATime.WaitAsync（）;
        尝试
        {
            resultProcessor（项目，结果）;
        }
        最后
        {
            oneAtATime.Release（）;
        }
    }
}

我并没有改变 ForEachAsync 的签名来选择并行的程度，我就让你希望你调整它。

输出例如：

  DownloadFileTaskAsync（空本地路径）：生成一个临时文件名http://cache.thephoenix.com/secure/uploadedImages/The_Phoenix/Music/CD_Review/main_OTR_Britney480.jpg
DownloadFileTaskAsync（空本地路径）：为http://ssimg.soundspike.com/artists/britneyspears_femmefatale_cd.jpg产生临时文件名
DownloadFileTaskAsync（空本地路径）：产生用于临时文件名http://a323.yahoofs.com/ymg/albumreviewsuk__1/albumreviewsuk-526650850-1301400550.jpg?ymm_1xEDE5bu0tMi
DownloadFileTaskAsync（空远程路径）：跳绳
DownloadFileTaskAsync（由于超时）：http://hangout.altsounds.com/geek/gars/images/3/9/8/5/2375.jpg
DownloadFileTaskAsync（由于超时）： http://www.beat.com.au/sites/default/files/imagecache/630_315sr/images/article/header/2011/april/britney-spears-femme-fatale.jpg
DownloadFileTaskAsync（由于超时）： http://cache.thephoenix.com/secure/uploadedImages/The_Phoenix/Music/CD_Review/main_OTR_Britney480.jpg
DownloadFileTaskAsync（下载）：http://newblog.thecmuwebsite.com/wp-content/uploads/2009/12/britneyspears1.jpg
DownloadFileTaskAsync（下载）：http://newblog.thecmuwebsite.com/wp-content/uploads/2009/12/britneyspears1.jpg
DownloadFileTaskAsync（下载）： http://static.guim.co.uk/sys-images/Music/Pix/site_furniture/2011/3/22/1300816812640/Femme-Fatale.jpg
DownloadFileTaskAsync（下载）：http://www.sputnikmusic.com/images/albums/72328.jpg

过去需要长达1分钟，现在几乎需要10秒相同的结果：）

和大感谢这2个职位作者：

<一个href=\"http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx\">http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx

<一个href=\"http://blogs.msdn.com/b/pfxteam/archive/2012/03/04/10277325.aspx\">http://blogs.msdn.com/b/pfxteam/archive/2012/03/04/10277325.aspx

EDIT

I've changed the title of the question to reflect the issue I had but also an answer on how to achieve this easily.

I am trying to make the 2nd method to return Task<TResult> instead of Task as in 1st method but I am getting a cascade of errors as a consequence of trying to fix it.

  • I added return before await body(partition.Current);
  • In turn it asks me to add a return statement below so I added return null below
  • But now the select statement complains that it cannot infer the type argument from the query
  • I change Task.Run to Task.Run<TResult> but without success.

How can I fix it ?

The first method comes from http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx, the second method is the overload that I'm trying to create.

public static class Extensions
{
    public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
    {
        return Task.WhenAll(
            from partition in Partitioner.Create(source).GetPartitions(dop)
            select Task.Run(async delegate
            {
                using (partition)
                    while (partition.MoveNext())
                        await body(partition.Current);
            }));
    }

    public static Task ForEachAsync<T, TResult>(this IEnumerable<T> source, int dop, Func<T, Task<TResult>> body)
    {
        return Task.WhenAll(
            from partition in Partitioner.Create(source).GetPartitions(dop)
            select Task.Run(async delegate
            {
                using (partition)
                    while (partition.MoveNext())
                        await body(partition.Current);
            }));
    }
}

Usage example :

With this method I'd like to download multiple files in parallel and asynchronously :

private async void MainWindow_Loaded(object sender, RoutedEventArgs e)
{
    Artist artist = await GetArtist();
    IEnumerable<string> enumerable = artist.Reviews.Select(s => s.ImageUrl);
    string[] downloadFile = await DownloadFiles(enumerable);
}

public static async Task<string[]> DownloadFiles(IEnumerable<string> enumerable)
{
    if (enumerable == null) throw new ArgumentNullException("enumerable");
    await enumerable.ForEachAsync(5, s => DownloadFile(s));
    // Incomplete, the above statement is void and can't be returned
}

public static async Task<string> DownloadFile(string address)
{
    /* Download a file from specified address, 
        * return destination file name on success or null on failure */

    if (address == null)
    {
        return null;
    }

    Uri result;
    if (!Uri.TryCreate(address, UriKind.Absolute, out result))
    {
        Debug.WriteLine(string.Format("Couldn't create URI from specified address: {0}", address));
        return null;
    }

    try
    {
        using (var client = new WebClient())
        {
            string fileName = Path.GetTempFileName();
            await client.DownloadFileTaskAsync(address, fileName);
            Debug.WriteLine(string.Format("Downloaded file saved to: {0} ({1})", fileName, address));
            return fileName;
        }
    }
    catch (WebException webException)
    {
        Debug.WriteLine(string.Format("Couldn't download file from specified address: {0}", webException.Message));
        return null;
    }
}

解决方案

I solved it and posting it here, might help anyone having the same issue.

My initial need was a small helper that would quickly download images but also just drop the connection if server does not respond quickly, all this in parallel and asynchronously.

This helper will return you a tuple that contains the remote path, the local path and the exception if one occurred; so quite useful as it's always good to know why faulty downloads have faulted. I think I forgot none of the situations that can occur for a download but you're welcome to comment it.

  • You specify a list of urls to download
  • You can specify a local file name where it will be saved, if not one will be generated for you
  • Optionally a duration for cancelling a download (handy for slow or unreachable servers)

You can just use DownloadFileTaskAsync itself or use the ForEachAsync helper for parallel and asynchronous downloads.

Code with an example on how to use it :

private async void MainWindow_Loaded(object sender, RoutedEventArgs e)
{
    IEnumerable<string> enumerable = your urls here;
    var results = new List<Tuple<string, string, Exception>>();
    await enumerable.ForEachAsync(s => DownloadFileTaskAsync(s, null, 1000), (url, t) => results.Add(t));
}

/// <summary>
///     Downloads a file from a specified Internet address.
/// </summary>
/// <param name="remotePath">Internet address of the file to download.</param>
/// <param name="localPath">
///     Local file name where to store the content of the download, if null a temporary file name will
///     be generated.
/// </param>
/// <param name="timeOut">Duration in miliseconds before cancelling the  operation.</param>
/// <returns>A tuple containing the remote path, the local path and an exception if one occurred.</returns>
private static async Task<Tuple<string, string, Exception>> DownloadFileTaskAsync(string remotePath,
    string localPath = null, int timeOut = 3000)
{
    try
    {
        if (remotePath == null)
        {
            Debug.WriteLine("DownloadFileTaskAsync (null remote path): skipping");
            throw new ArgumentNullException("remotePath");
        }

        if (localPath == null)
        {
            Debug.WriteLine(
                string.Format(
                    "DownloadFileTaskAsync (null local path): generating a temporary file name for {0}",
                    remotePath));
            localPath = Path.GetTempFileName();
        }

        using (var client = new WebClient())
        {
            TimerCallback timerCallback = c =>
            {
                var webClient = (WebClient) c;
                if (!webClient.IsBusy) return;
                webClient.CancelAsync();
                Debug.WriteLine(string.Format("DownloadFileTaskAsync (time out due): {0}", remotePath));
            };
            using (var timer = new Timer(timerCallback, client, timeOut, Timeout.Infinite))
            {
                await client.DownloadFileTaskAsync(remotePath, localPath);
            }
            Debug.WriteLine(string.Format("DownloadFileTaskAsync (downloaded): {0}", remotePath));
            return new Tuple<string, string, Exception>(remotePath, localPath, null);
        }
    }
    catch (Exception ex)
    {
        return new Tuple<string, string, Exception>(remotePath, null, ex);
    }
}

public static class Extensions
{
    public static Task ForEachAsync<TSource, TResult>(
        this IEnumerable<TSource> source,
        Func<TSource, Task<TResult>> taskSelector, Action<TSource, TResult> resultProcessor)
    {
        var oneAtATime = new SemaphoreSlim(5, 10);
        return Task.WhenAll(
            from item in source
            select ProcessAsync(item, taskSelector, resultProcessor, oneAtATime));
    }

    private static async Task ProcessAsync<TSource, TResult>(
        TSource item,
        Func<TSource, Task<TResult>> taskSelector, Action<TSource, TResult> resultProcessor,
        SemaphoreSlim oneAtATime)
    {
        TResult result = await taskSelector(item);
        await oneAtATime.WaitAsync();
        try
        {
            resultProcessor(item, result);
        }
        finally
        {
            oneAtATime.Release();
        }
    }
}

I haven't changed the signature of ForEachAsync to choose the level of parallelism, I'll let you adjust it as you wish.

Output example : 

DownloadFileTaskAsync (null local path): generating a temporary file name for http://cache.thephoenix.com/secure/uploadedImages/The_Phoenix/Music/CD_Review/main_OTR_Britney480.jpg
DownloadFileTaskAsync (null local path): generating a temporary file name for http://ssimg.soundspike.com/artists/britneyspears_femmefatale_cd.jpg
DownloadFileTaskAsync (null local path): generating a temporary file name for http://a323.yahoofs.com/ymg/albumreviewsuk__1/albumreviewsuk-526650850-1301400550.jpg?ymm_1xEDE5bu0tMi
DownloadFileTaskAsync (null remote path): skipping
DownloadFileTaskAsync (time out due): http://hangout.altsounds.com/geek/gars/images/3/9/8/5/2375.jpg
DownloadFileTaskAsync (time out due): http://www.beat.com.au/sites/default/files/imagecache/630_315sr/images/article/header/2011/april/britney-spears-femme-fatale.jpg
DownloadFileTaskAsync (time out due): http://cache.thephoenix.com/secure/uploadedImages/The_Phoenix/Music/CD_Review/main_OTR_Britney480.jpg
DownloadFileTaskAsync (downloaded): http://newblog.thecmuwebsite.com/wp-content/uploads/2009/12/britneyspears1.jpg
DownloadFileTaskAsync (downloaded): http://newblog.thecmuwebsite.com/wp-content/uploads/2009/12/britneyspears1.jpg
DownloadFileTaskAsync (downloaded): http://static.guim.co.uk/sys-images/Music/Pix/site_furniture/2011/3/22/1300816812640/Femme-Fatale.jpg
DownloadFileTaskAsync (downloaded): http://www.sputnikmusic.com/images/albums/72328.jpg

What used to take up to 1 minute now barely takes 10 seconds for the same result :)

And big thanks to the author of these 2 posts :

http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx

http://blogs.msdn.com/b/pfxteam/archive/2012/03/04/10277325.aspx

这篇关于异步和并行下载文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
C#/.NET最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆