网络爬虫评测? [英] Web crawling evaluation?

问题描述

我在专注网络爬行(又名主题网络爬行)中看到，评估指标 - 收获率 - 定义为:在爬行 't' 个页面后，收获率 = number_of_relevant_pages/pages_crawled(t).

I have seen in focused web crawling (a.k.a. topical web crawling), the evaluation metric - harvest ratio - is defined as: after crawling 't' pages, harvest ratio = number_of_relevant_pages/pages_crawled(t).

例如，在抓取 100 个页面后，我得到了 80 个真阳性，那么此时抓取工具的收获率为 0.9.但是爬虫可能会忽略一些与爬行域完全相关但未计入评估比率的页面.这是什么?我们能否改进该评估指标以包含完全相关的遗漏页面?这个考虑重要吗?

So for example after crawling 100 pages I get 80 true positives then the harvest ratio of the crawler at that point is 0.9. But the crawler might have ignored some pages off crawling that are totally relevant to the crawling domain but is not accounted in the evaluation ratio. What is this? Can we improve that evaluation metric to include the missed pages that are totally relevant? Is this consideration important?

推荐答案

聚焦爬行最基本的评估是精度和召回率，它们可以聚合到 F-measure 中.

The most basic evaluation for a focused crawl is Precision and recall which can be aggregated into F-measure.

http://en.wikipedia.org/wiki/Precision_and_recall

http://en.wikipedia.org/wiki/F1_score

如果您对页面与特定关键字的相关性更感兴趣，您可以使用 tf/idf(词频-逆文档频率)

If you are more interested into how a page is relevant to a specific keyword, you want to use tf/idf (term frequency–inverse document frequency)

http://en.wikipedia.org/wiki/Tf*idf

这篇关于网络爬虫评测?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！