仅生成未提取的网址，而不是获得评分的Nutch 2.3 [英] Generate only unfetched urls instead of scored Nutch 2.3

问题描述

有什么方法可以只生成未提取的网址，而不是基于Nutch 2.x中的分数吗?

Is there any way to generate only the un-fetched urls instead of based on score in Nutch 2.x?

推荐答案

好吧，对于Nutch 1.x，您可以使用自Nutch 1.12(我认为)以来提供的jexl支持:

Well, for Nutch 1.x you could use the jexl support that is shipped since Nutch 1.12 (I think):

$ bin/nutch generate -expr "status == db_unfetched" 

使用此命令，您可以确保仅考虑具有db_unfetched状态的URL来生成要爬网的细分.

with this command you're ensuring that only the URLs with a db_unfetched status are considered for generating the segments that you want to crawl.

此功能在2.x分支上仍然不可用，但是编写自定义GeneratorJob可以解决问题.

This feature is still not available on 2.x branch, but writing a custom GeneratorJob could do the trick.

另一方面，由于生成器Job已在考虑分数以对要获取的URL列表进行排序，因此可能更简单的方法是编写自定义

On the other hand, since the generator Job is already considering the score to sort the list of URLs to fetch, perhaps the easier way could be to write a custom ScoringFilter.

例如，如果您看一下:

For instance, if you take a look at: https://github.com/apache/nutch/blob/2.x/src/java/org/apache/nutch/scoring/ScoringFilter.java#L69-L81 the ScoringFilter already provides a generatorSortValue method only for generating a scoring value for the generator job, so you could write your own to boost those URLs with an unfetched status.

这篇关于仅生成未提取的网址，而不是获得评分的Nutch 2.3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！