Nutch 不会抓取多个站点 [英] Nutch does not crawl multiple sites
本文介绍了Nutch 不会抓取多个站点的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试使用 Nutch 抓取多个网站.我的 seed.txt 看起来像这样:
I'm trying to crawl multiple sites using Nutch. My seed.txt looks like this:
http://1.a.b/
http://2.a.b/
我的 regex-urlfilter.txt 看起来像这样:
and my regex-urlfilter.txt looks like this:
# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
# for a more extensive coverage use the urlfilter-suffix plugin
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
#+.
+^http://1.a.b/*
+^http://2.a.b/*
我在最后一部分尝试了以下内容:
I tried the following for the last part:
+^http://([a-z0-9]*\.)*a.b/*
唯一被抓取的网站是第一个.所有其他配置都是默认的.
The only site crawled is the first one. All other configuration is default.
我运行以下命令:
bin/nutch crawl urls -solr http://localhost:8984/solr/ -dir crawl -depth 10 -topN 10
有什么想法吗?!
谢谢!
推荐答案
在 regex-urlfilter.txt 中试试这个:
Try this in regex-urlfilter.txt :
旧设置:
# accept anything else
#+.
+^http://1.a.b/*
+^http://2.a.b/*
新插入:
# accept anything else
+.
这篇关于Nutch 不会抓取多个站点的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文