如何使用 xmlfeedspider 抓取 xml 提要 [英] How to scrape xml feed with xmlfeedspider
本文介绍了如何使用 xmlfeedspider 抓取 xml 提要的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试使用以下格式抓取 xml 文件
I am trying to scrape an xml file with the below format
file_sample.xml:
<rss version="2.0">
<channel>
<item>
<title>SENIOR BUDGET ANALYST (new)</title>
<link>https://hr.example.org/psp/hrapp&SeqId=1</link>
<pubDate>Wed, 18 Jul 2012 04:00:00 GMT</pubDate>
<category>All Open Jobs</category>
</item>
<item>
<title>BUDGET ANALYST (healthcare)</title>
<link>https://hr.example.org/psp/hrapp&SeqId=2</link>
<pubDate>Wed, 18 Jul 2012 04:00:00 GMT</pubDate>
<category>All category</category>
</item>
</channel>
</rss>
下面是我的spider.py代码
class TestSpider(XMLFeedSpider):
name = "testproject"
allowed_domains = {"www.example.com"}
start_urls = [
"https://www.example.com/hrapp/rss/careers_jo_rss.xml"
]
iterator = 'iternodes'
itertag = 'channel'
def parse_node(self, response, node):
title = node.select('item/title/text()').extract()
link = node.select('item/link/text()').extract()
pubdate = node.select('item/pubDate/text()').extract()
category = node.select('item/category/text()').extract()
item = TestprojectItem()
item['title'] = title
item['link'] = link
item['pubdate'] = pubdate
item['category'] = category
return item
结果:
2012-07-25 13:24:14+0530 [testproject] DEBUG: Scraped from <200 https://hr.templehealth.org/hrapp/rss/careers_jo_rss.xml>
{'title': [u'SENIOR BUDGET ANALYST (hospital/healthcare)',
u'BUDGET ANALYST'],
'link': [u'https://hr.example.org/psp/hrapp&SeqId=1',
u'https://hr.example.org/psp/hrapp&SeqId=2']
'pubdate': [u'Wed, 18 Jul 2012 04:00:00 GMT',
u'Wed, 18 Jul 2012 04:00:00 GMT']
'category': [u'All Open Jobs',
u'All category']
}
在这里你可以从上面的结果中观察到,来自相应标签的所有结果都被组合到一个列表中,但我想根据它们的单个项目标签进行映射,如下所示,因为我们为 html 抓取而这样做.>
here as u can observe from the above result, all the results from the corresponding tags are combined in to single list, but i want to map according to their individual item tag like below as we do it for html scraping.
{'title': u'SENIOR BUDGET ANALYST (hospital/healthcare)'
'link': u'https://hr.example.org/psp/hrapp&SeqId=1'
'pubdate': u'Wed, 18 Jul 2012 04:00:00 GMT'
'category': u'All Open Jobs'
}
{'title': u'BUDGET ANALYST'
'link': u'https://hr.example.org/psp/hrapp&SeqId=2'
'pubdate': u'Wed, 18 Jul 2012 04:00:00 GMT'
'category': u'All category'
}
我们如何根据单独的主标签(如上面的 item 标签)抓取 xml 标签数据.
How can we scrape xml tag data according to separate main tag like item tag above.
提前致谢.................
Thanks in advance.............
推荐答案
我推荐使用 feedparser:
feedparser.parse(url)
结果
{'bozo': 1,
'bozo_exception': xml.sax._exceptions.SAXParseException("EntityRef: expecting ';'\n"),
'encoding': u'utf-8',
'entries': [{'link': u'https://hr.example.org/psp/hrapp&SeqId=1',
'links': [{'href': u'https://hr.example.org/psp/hrapp&SeqId=1',
'rel': u'alternate',
'type': u'text/html'}],
'tags': [{'label': None, 'scheme': None, 'term': u'All Open Jobs'}],
'title': u'SENIOR BUDGET ANALYST (new)',
'title_detail': {'base': u'',
'language': None,
'type': u'text/plain',
'value': u'SENIOR BUDGET ANALYST (new)'},
'updated': u'Wed, 18 Jul 2012 04:00:00 GMT',
'updated_parsed': time.struct_time(tm_year=2012, tm_mon=7, tm_mday=18, tm_hour=4, tm_min=0, tm_sec=0, tm_wday=2, tm_yday=200, tm_isdst=0)},
{'link': u'https://hr.example.org/psp/hrapp&SeqId=2',
'links': [{'href': u'https://hr.example.org/psp/hrapp&SeqId=2',
'rel': u'alternate',
'type': u'text/html'}],
'tags': [{'label': None, 'scheme': None, 'term': u'All category'}],
'title': u'BUDGET ANALYST (healthcare)',
'title_detail': {'base': u'',
'language': None,
'type': u'text/plain',
'value': u'BUDGET ANALYST (healthcare)'},
'updated': u'Wed, 18 Jul 2012 04:00:00 GMT',
'updated_parsed': time.struct_time(tm_year=2012, tm_mon=7, tm_mday=18, tm_hour=4, tm_min=0, tm_sec=0, tm_wday=2, tm_yday=200, tm_isdst=0)}],
'feed': {},
'namespaces': {},
'version': u'rss20'}
这篇关于如何使用 xmlfeedspider 抓取 xml 提要的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文