从净使用线程下载文件或不 [英] Download files from net using threading or not

问题描述

我需要下载一个巨大的基于关键字净文件数量。 我下面的步骤

1. 使用刮弄明白的链接，文件
2. 使用WebClient.DownloadData（）下载字节[]
3. 保存ARR到一个文件中。

这是个好主意，以创建一个线程下载各个文件获得更好的性能。 有什么建议么。 谢谢

 的foreach（字符串在每个ARR）
        {

            线程t =新主题（
                                新的ThreadStart（
                                    代表
                                    {

                                        字符串[] ARR2 = each.Split（新的String [] {HTTP：//}，StringSplitOptions.None）;

                                        字符串[] firstElem = ARR2 [1] .Split（新的String [] {}，StringSplitOptions.None）;

                                        串urlToDownload = @firstElem [0] .Replace（\，的String.Empty）;
                                        字符串FILNAME = Path.GetFileName（urlToDownload）;
                                        字符串目录名= DirInAppConfig（）;
                                        DataRow的列;
                                        布尔dataExistsInDtKwWithSameDownloadLinkAndFileName;
                                        getRowForKwDownLinkFileName（urlToDownload，FILNAME，出列，出dataExistsInDtKwWithSameDownloadLinkAndFileName）;
                                        downloadFile（客户端，urlToDownload，目录名，FILNAME，搜索，行）;
                                    }））;
                                t.IsBackground = TRUE;
                                t.Start（）;
                                t.Join（）;
        }
 

解决方案

通常服务器限制下载从一个IP到2个连接。因此，如果所有文件都来自同一服务器，多个线程可能没有多大帮助。

I need to download a huge number of files from net based on a keyword. The steps i am following are

1. Using Scraping figure out the links to files
2. Using WebClient.DownloadData() download the byte[]
3. Save the arr to a file.

Is it a good idea to create one thread for downloading each file for better performance. Any suggestions. Thanks

foreach (string each in arr)
        {

            Thread t = new Thread(
                                new ThreadStart(
                                    delegate
                                    {

                                        string[] arr2 = each.Split(new string[] { "http://" }, StringSplitOptions.None);

                                        string[] firstElem = arr2[1].Split(new string[] { " " }, StringSplitOptions.None);

                                        string urlToDownload = @firstElem[0].Replace("\"", string.Empty);
                                        string filName = Path.GetFileName(urlToDownload);
                                        string dirName = DirInAppConfig();
                                        DataRow row;
                                        bool dataExistsInDtKwWithSameDownloadLinkAndFileName;
                                        getRowForKwDownLinkFileName(urlToDownload, filName, out row, out dataExistsInDtKwWithSameDownloadLinkAndFileName);
                                        downloadFile(Client, urlToDownload, dirName, filName, search, row);
                                    }));
                                t.IsBackground = true;
                                t.Start();
                                t.Join();
        }

解决方案

Often server limit the download from one IP to 2 connections. So if all files are from the same server, multiple threads might not help much.

这篇关于从净使用线程下载文件或不的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！