有没有办法确定理想的线程数？ [英] Is there a way to determine the ideal number of threads?

问题描述

我正在做一个webcrawler并使用线程下载页面。

I am doing a webcrawler and using threads to download pages.

我程序性能的第一个限制因素是带宽，我永远无法下载更多页面它可以得到。

The first limiting factor to the performance of my program is the bandwidth, I can never download more pages that it can get.

第二件事是我感兴趣的。我正在使用线程同时下载许多页面，但是当我创建更多线程时，会发生更多的处理器共享。是否有一些度量/方式/类测试来确定什么是理想的线程数，或者如果在一定数量之后，性能不会改变或减少？

The second thing is what I interested. I am using threads to download many pages at same time, but as I create more threads, more sharing of processor occurs. Is there some metric/way/class of tests to determine what is the ideal number of threads or if after certain number, the performance doesn't change or decrease?

推荐答案

我们开发了一个多线程并行网络爬虫。对吞吐量进行基准测试是了解野兽如何处理其工作的最佳方式。对于专用的java服务器，每个核心一个线程是一个基础，然后I / O开始发挥作用。

we've developped a multithreaded parrallel web crawler. Benchmarking troughput is the best way to get ideas on how the beast will handle his job. For a dedicated java server, one thread per core is a base to start, then the I/O comes into play and change.

在一定数量的线程之后性能会降低。但这取决于你抓取的网站，你使用的操作系统等等。尝试找到一个只有一个恒定响应时间的网站来做你的第一个基准测试（比如谷歌，但采取不同的服务）

Performances do decrease after certain number of threads. But it depends on the site you crawl too, on the OS you use, etc. Try to find a site with a merely constant response time to do your first benchmarks (like Google, but take differents services)

网站速度慢，线程数量越多，补偿i / o阻塞

With slow websites, higher number of threads tends to compensate i/o blocking

这篇关于有没有办法确定理想的线程数？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！