如何让我的pogram在多个线程上运行 [英] How to make my pogram run on multiple threads

问题描述

所以我正在制作一个网络刮刀，但它有点慢，有些人指出我应该让我的程序在多个线程上运行来修复问题。所以我想做的是制作相同的程序运行100+ threads.Can有人帮我做，如果需要，将分享
源代码

 class Program 
 {
 private static String _outputFileAvailable =" available.txt" ;; 
 private static String _outputFileUnavailable =" unavailable.txt" ;; 
 
 
 static void Main（string [] args）
 {
 
 Console.WriteLine（" Proxies Type（HTTP / SOCKS4 / SOCKS5）： "）; 
 string proxiesType = Console.ReadLine（）; 
 var AvailableNum = 0; 
 var UnavailableNum = 0; 
 Console.Title =" Cookie名称检查器|由BataBo |检查：" +（AvailableNum + UnavailableNum）+" |良好：" + AvailableNum; 
 using（StreamWriter availableWriter = File.AppendText（_outputFileUnavailable））
 {
 using（StreamWriter unavailableWriter = File.AppendText（_outputFileAvailable））
 {
 foreach（string line在File.ReadLines（" Usernames.txt"））
 {
 args = new [] 
 {
" https：//api.mojang.com/users/轮廓/的Minecraft /" + line 
}; 
 
 var plines = File.ReadAllLines（" Proxies.txt"）; 
 var prline = new Random（）; 
 var prnline = prline.Next（0，plines.Length  -  1）; 
 var pline = plines [prnline]; 
 WebProxy proxy = new WebProxy（proxiesType +"：//" + pline +" /"）; 
 
 
 var client = new WebClient（）; 
 
 client.Proxy = proxy; 
 
 
 client.Headers [HttpRequestHeader.UserAgent] =" Mozilla / 5.0（Windows NT 6.1）AppleWebKit / 537.11（KHTML，类似Gecko）Chrome / 23.0.1271.97 Safari / 537.11" ; 
 client.Headers [HttpRequestHeader.ContentType] =" application / x-www-form-urlencoded" ;; 
 client.Headers [HttpRequestHeader.Accept] =" text / html，application / xhtml + xml，application / xml; q = 0.9，* / *; q = 0.8" ;; 
 client.Headers [HttpRequestHeader.AcceptEncoding] =" gzip，deflate，sdch" ;; 
 client.Headers [HttpRequestHeader.AcceptLanguage] =" en-GB，en-US; q = 0.8，en; q = 0.6" ;; 
 client.Headers [HttpRequestHeader.AcceptCharset] =" ISO-8859-1，utf-8; q = 0.7，*; q = 0.3" ;; 
 
 var content = ScrubContent（client.DownloadString（args [0]））; 
 
 
 if（content ==""）
 {
 Console.ForegroundColor = ConsoleColor.Green; 
 Console.WriteLine（"名称{0}可用"，行）; 
 unavailableWriter.WriteLine（line）; 
 AvailableNum ++; 
 Console.Title =" Cookie名称检查器|由BataBo |检查：" +（AvailableNum + UnavailableNum）+" |良好：" + AvailableNum; 
} 
 
 else 
 {
 Console.ForegroundColor = ConsoleColor.Red; 
 Console.WriteLine（"名称{0}不可用"，行）; 
 availableWriter.WriteLine（line）; 
 UnavailableNum ++; 
 Console.Title =" Cookie名称检查器|由BataBo |检查：" +（AvailableNum + UnavailableNum）+" |良好：" + AvailableNum; 
} 
} 
} 
} 
 Console.WriteLine（""）; 
 Console.ForegroundColor = ConsoleColor.Yellow; 
 Console.WriteLine（" ==================== Results =================== ="）; 
 Console.ForegroundColor = ConsoleColor.Blue; 
 Console.WriteLine（" Checked：{0} names"，AvailableNum + UnavailableNum）; 
 Console.ForegroundColor = ConsoleColor.Green; 
 Console.WriteLine（" Available names：{0} names"，AvailableNum）; 
 Console.ForegroundColor = ConsoleColor.Red; 
 Console.WriteLine（" Unavailable names：{0} names"，UnavailableNum）; 
 Console.ForegroundColor = ConsoleColor.Yellow; 
 Console.WriteLine（" ==================== Results =================== ="）; 
 Console.ReadLine（）; 
} 
静态字符串ScrubContent（字符串内容）
 {
返回新字符串（content.Where（c => c！='\ n'）。ToArray（） ）; 
} 
 
} 

解决方案

在决定在多个线程上运行程序之前，首先必须找出导致它"有点慢"的瓶颈。

• 如果瓶颈在您的网络带宽中，那么启动多个线程就无法获得任何收益。
• 如果瓶颈在于远程计算机的响应速度，那么报废（假设你是从不同的站点报废）然后为每个服务器启动一个不同的线程可能是有利的。
• 如果在处理你报废的数据时瓶颈是你的CPU速度，那么制作程序在100个线程上运行可能是个坏主意，除非你在具有100个内核的计算机上运行它。不要启动超过CPU数量的线程。

视具体情况而定，以及程序如何处理数据，您启动多个线程的方式将有所不同。对于"cpu" case，最简单的方法可能是执行一个Parallel.Foreach循环，迭代你要处理的所有
文档。

So I'm making a web scraper but its kinda slow and some people pointed out that I should make my program run on multiple threads to fix the issue.So what I want to do is to make the same program run on 100+ threads.Can somebody help me do it,will share source code if necessery

 class Program
    {
        private static String _outputFileAvailable = "available.txt";
        private static String _outputFileUnavailable = "unavailable.txt";


        static void Main(string[] args)
        {
            
            Console.WriteLine("Proxies Type (HTTP/SOCKS4/SOCKS5) : ");
            string proxiesType = Console.ReadLine();
            var AvailableNum = 0;
            var UnavailableNum = 0;
            Console.Title = "Cookie name checker | by BataBo | Checked: " + (AvailableNum + UnavailableNum) +  " | Good:" + AvailableNum;
            using (StreamWriter availableWriter = File.AppendText(_outputFileUnavailable))
            {
                using (StreamWriter unavailableWriter = File.AppendText(_outputFileAvailable))
                {
                    foreach (string line in File.ReadLines("Usernames.txt"))
                    {
                        args = new[]
                        {
                "https://api.mojang.com/users/profiles/minecraft/" + line
            };
                        
                        var plines = File.ReadAllLines("Proxies.txt");
                        var prline = new Random();
                        var prnline = prline.Next(0, plines.Length - 1);
                        var pline = plines[prnline];
                        WebProxy proxy = new WebProxy(proxiesType + "://" + pline + "/");
                        

                        var client = new WebClient();

                        client.Proxy = proxy;
                        

                        client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11";
                        client.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
                        client.Headers[HttpRequestHeader.Accept] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
                        client.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate,sdch";
                        client.Headers[HttpRequestHeader.AcceptLanguage] = "en-GB,en-US;q=0.8,en;q=0.6";
                        client.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";

                        var content = ScrubContent(client.DownloadString(args[0]));


                        if (content == "")
                        {
                            Console.ForegroundColor = ConsoleColor.Green;
                            Console.WriteLine("Name {0} is available", line);
                            unavailableWriter.WriteLine(line);
                            AvailableNum++;
                            Console.Title = "Cookie name checker | by BataBo | Checked: " + (AvailableNum + UnavailableNum)  + " | Good:" + AvailableNum;
                        }

                        else
                        {
                            Console.ForegroundColor = ConsoleColor.Red;
                            Console.WriteLine("Name {0} is not available", line);
                            availableWriter.WriteLine(line);
                            UnavailableNum++;
                            Console.Title = "Cookie name checker | by BataBo | Checked: " + (AvailableNum + UnavailableNum) + " | Good:" + AvailableNum;
                        }
                    }
                }
            }
            Console.WriteLine("");
            Console.ForegroundColor = ConsoleColor.Yellow;
            Console.WriteLine("====================Results====================");
            Console.ForegroundColor = ConsoleColor.Blue;
            Console.WriteLine("Checked: {0} names", AvailableNum + UnavailableNum);
            Console.ForegroundColor = ConsoleColor.Green;
            Console.WriteLine("Available names: {0} names", AvailableNum);
            Console.ForegroundColor = ConsoleColor.Red;
            Console.WriteLine("Unavailable names: {0} names", UnavailableNum);
            Console.ForegroundColor = ConsoleColor.Yellow;
            Console.WriteLine("====================Results====================");
            Console.ReadLine();
        }
            static string ScrubContent(string content)
            {
                return new string(content.Where(c => c != '\n').ToArray());
            }

        }

解决方案

Before deciding to run the program on multiple threads, you first have to find out what is the bottleneck that makes it "kinda slow".

• If the bottleneck is in your network bandwidth, then you will not gain anything from launching multiple threads.
• If the bottleneck is in the speed of response of the remote computers that you are scrapping (presuming you are scrapping from different sites) then it may be advantageous to launch a different thread for each server.
• If the bottleneck is in your CPU speed when processing the data that you have scrapped, then making the program run on 100 threads is probably a bad idea, unless you run it on a computer with 100 cores. Don't launch more threads than the number of CPUs.

Depending on the case, and how your program is processing its data, the way in which you will launch your multiple threads will be different. For the "cpu" case, the simplest way is probably to do a Parallel.Foreach loop that iterates on all the documents that you want to process.

这篇关于如何让我的pogram在多个线程上运行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！