使用 Scrapy 抓取本地 XML 文件 - 起始 URL 本地文件地址 [英] Use Scrapy to crawl local XML file - Start URL local file address
本文介绍了使用 Scrapy 抓取本地 XML 文件 - 起始 URL 本地文件地址的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想用scrapy抓取我在下载文件夹中找到的本地xml文件,使用xpath提取相关信息.
I want to crawl a local xml file that I have located in my Downloads folder with scrapy, use xpath to extract the relevant information.
使用scrapy 介绍作为指南
Using the scrapy intro as a guide
2016-01-24 12:38:53 [scrapy] DEBUG: Retrying <GET file://home/sayth/Downloads/20160123RAND0.xml> (failed 2 times): [Errno 2] No such file or directory: '/sayth/Downloads/20160123RAND0.xml'
2016-01-24 12:38:53 [scrapy] DEBUG: Gave up retrying <GET file://home/sayth/Downloads/20160123RAND0.xml> (failed 3 times): [Errno 2] No such file or directory: '/sayth/Downloads/20160123RAND0.xml'
2016-01-24 12:38:53 [scrapy] ERROR: Error downloading <GET file://home/sayth/Downloads/20160123RAND0.xml>
我已经尝试了以下几个版本,但是我无法获得开始 URL 来接受我的文件.
I have tried several version of the below however i am not able to get the start url to accept my file.
# -*- coding: utf-8 -*-
import scrapy
class MyxmlSpider(scrapy.Spider):
name = "myxml"
allowed_domains = ["file://home/sayth/Downloads"]
start_urls = (
'http://www.file://home/sayth/Downloads/20160123RAND0.xml',
)
def parse(self, response):
for file in response.xpath('//meeting'):
full_url = response.urljoin(href.extract())
yield scrapy.Request(full_url, callback=self.parse_question)
def parse_xml(self, response):
yield {
'name': response.xpath('//meeting/race').extract()
}
只是为了确认我确实在那个位置有文件
Just to confirm I do have the file in that location
sayth@sayth-HP-EliteBook-2560p : ~/Downloads
[0] % ls -a
. Building a Responsive Website with Bootstrap [Video].zip
.. codemirror.zip
1.1 Situation Of Long Term Gain.xls Complete-Python-Bootcamp-master.zip
2008 Racedata.xls Cox Plate 2005.xls
20160123RAND0.xml
推荐答案
完全不要指定 allowed_domains
并在协议后使用 3 个斜杠:
Don't specify the allowed_domains
at all and use 3 slashes after the protocol:
start_urls = ["file:///home/sayth/Downloads/20160123RAND0.xml"]
这篇关于使用 Scrapy 抓取本地 XML 文件 - 起始 URL 本地文件地址的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文