如何使用Scrapy抓取网站所有页面上的链接 [英] How to crawl links on all pages of a web site with Scrapy
问题描述
我正在学习scrapy，并且尝试提取包含以下内容的所有链接:" http://lattes .cnpq.br/andasequenceofnumbers "，例如:http://lattes.cnpq.br/0281123427918302
但是我不知道网站上包含这些信息的页面是什么.
例如，此网站:
I'm learning about scrapy and I'm trying to extract all links that contains: "http://lattes.cnpq.br/andasequenceofnumbers" , example: http://lattes.cnpq.br/0281123427918302
But I don't know what is the page on the web site that contains these information.
For example this web site:
http://www.ppgcc.ufv.br/
我想要的链接在此页面上:
The links that I want are on this page:
http://www.ppgcc.ufv.br/?page_id=697
我该怎么办? 我正在尝试使用规则，但是我不知道如何正确使用正则表达式. 谢谢
What could I do? I'm trying to use rules but I don't know how to use regular expressions correctly. Thank you
我需要在主站点(ppgcc.ufv.br)的所有页面上搜索链接(http://lattes.cnpq.br/asequenceofnumber)的类型.我的目标是获取所有链接lattes.cnpq.br/numbers，但我不知道它们在哪里.我正在使用一个简单的代码，实际上是这样的:
I need search on all pages of the main (ppgcc.ufv.br) site the kind of links (http://lattes.cnpq.br/asequenceofnumber). My Objective is get all the links lattes.cnpq.br/numbers but I don't know where they are. I'm using a simple code actually like:
class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["ppgcc.ufv.br"]
start_urls = (
'http://www.ppgcc.ufv.br/',
)
rules = [Rule(SgmlLinkExtractor(allow=[r'.*']), follow=True),
Rule(SgmlLinkExtractor(allow=[r'@href']), callback='parse')]
def parse(self, response):
filename = str(random.randint(1, 9999))
open(filename, 'wb').write(response.body)
#I'm trying to understand how to use rules correctly
2 EDIT ----
使用:
class ExampleSpider(CrawlSpider):
name = "example"
allowed_domains = [".ppgcc.ufv.br"]
start_urls = (
'http://www.ppgcc.ufv.br/',
)
rules = [Rule(SgmlLinkExtractor(allow=[r'.*']), follow=True),
Rule(SgmlLinkExtractor(allow=[r'@href']), callback='parse_links')]
def parse_links(self, response):
filename = "Lattes.txt"
arquivo = open(filename, 'wb')
extractor = LinkExtractor(allow=r'lattes\.cnpq\.br/\d+')
for link in extractor.extract_links(response):
url = link.urlextractor = LinkExtractor(allow=r'lattes\.cnpq\.br/\d+')
arquivo.writelines("%s\n" % url)
print url
它告诉我:
C:\Python27\Scripts\tutorial3>scrapy crawl example
2015-06-02 08:08:18-0300 [scrapy] INFO: Scrapy 0.24.6 started (bot: tutorial3)
2015-06-02 08:08:18-0300 [scrapy] INFO: Optional features available: ssl, http11
2015-06-02 08:08:18-0300 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial3.spiders', 'SPIDER_MODULES': ['tutorial3
.spiders'], 'BOT_NAME': 'tutorial3'}
2015-06-02 08:08:19-0300 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-06-02 08:08:19-0300 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMidd
leware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMidd
leware, ChunkedTransferMiddleware, DownloaderStats
2015-06-02 08:08:19-0300 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLe
ngthMiddleware, DepthMiddleware
2015-06-02 08:08:19-0300 [scrapy] INFO: Enabled item pipelines:
2015-06-02 08:08:19-0300 [example] INFO: Spider opened
2015-06-02 08:08:19-0300 [example] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-06-02 08:08:19-0300 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-06-02 08:08:19-0300 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2015-06-02 08:08:19-0300 [example] DEBUG: Crawled (200) <GET http://www.ppgcc.ufv.br/> (referer: None)
2015-06-02 08:08:19-0300 [example] DEBUG: Filtered offsite request to 'www.cgu.gov.br': <GET http://www.cgu.gov.br/acessoainformacao
gov/>
2015-06-02 08:08:19-0300 [example] DEBUG: Filtered offsite request to 'www.brasil.gov.br': <GET http://www.brasil.gov.br/>
2015-06-02 08:08:19-0300 [example] DEBUG: Filtered offsite request to 'www.ppgcc.ufv.br': <GET http://www.ppgcc.ufv.br/>
2015-06-02 08:08:19-0300 [example] DEBUG: Filtered offsite request to 'www.ufv.br': <GET http://www.ufv.br/>
2015-06-02 08:08:19-0300 [example] DEBUG: Filtered offsite request to 'www.dpi.ufv.br': <GET http://www.dpi.ufv.br/>
2015-06-02 08:08:19-0300 [example] DEBUG: Filtered offsite request to 'www.portal.ufv.br': <GET http://www.portal.ufv.br/?page_id=84
>
2015-06-02 08:08:19-0300 [example] DEBUG: Filtered offsite request to 'www.wordpress.org': <GET http://www.wordpress.org/>
2015-06-02 08:08:19-0300 [example] INFO: Closing spider (finished)
2015-06-02 08:08:19-0300 [example] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 215,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 18296,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2015, 6, 2, 11, 8, 19, 912000),
'log_count/DEBUG': 10,
'log_count/INFO': 7,
'offsite/domains': 7,
'offsite/filtered': 42,
'request_depth_max': 1,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2015, 6, 2, 11, 8, 19, 528000)}
2015-06-02 08:08:19-0300 [example] INFO: Spider closed (finished)
我一直在寻找网站的源代码，爬网没有获取的页面链接更多，也许我的规则不正确
And I was looking the source code of the site, there are more links of pages that the crawl didn't GET, maybe my rules are incorrect
推荐答案
因此，首先要注意以下几点:
So, a couple things first:
1)rules属性仅在扩展CrawlSpider类时有效，而在扩展简单的scrapy.Spider时则无效.
1) the rules attribute only works if you're extending the CrawlSpider class, they won't work if you extend the simpler scrapy.Spider.
2)如果您使用rules和CrawlSpider路由，则不应覆盖默认的parse回调，因为默认实现实际上是调用规则的方式-因此，您想使用其他名称回调.
2) if you go the rules and CrawlSpider route, you should not override the default parse callback, because the default implementation is what actually calls the rules -- so you want to use another name for your callback.
3) to do the actual extraction of the links you want, you can use a LinkExtractor inside your callback to scrape the links from the page:
from scrapy.contrib.linkextractors import LinkExtractor
class MySpider(scrapy.Spider):
...
def parse_links(self, response):
extractor = LinkExtractor(allow=r'lattes\.cnpq\.br/\d+')
for link in extractor.extract_links(response):
item = LattesItem()
item['url'] = link.url
希望对您有帮助.
这篇关于如何使用Scrapy抓取网站所有页面上的链接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
