Xpath() 方法不使用 Scrapy 返回结果 [英] Xpath() method not returning results using Scrapy

问题描述

我在 Windows Vista 64 位上使用 Python.org 版本 2.7 64 位.我有一些 Scrapy 代码试图解析此链接中的表格，标题为韦恩鲁尼的比赛历史:"...

I am using Python.org version 2.7 64 bit on Windows Vista 64 bit. I have some Scrapy code that is attempting to parse the table at this link headed 'Wayne Rooney's Match History:"...

http://www.whoscored.com/Players/3859/Fixtures/韦恩-鲁尼我到目前为止的代码是这样的:

http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney The code I have so far is this:

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re


class MySpider(Spider):
    name = "wiki"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]

    def parse(self, response):

        for row in response.selector.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]'):
        # Is this row contains goal symbols?
            list_of_goals = row.xpath('//span[@title="Goal"]')
            if list_of_goals:
                list = str(list_of_goals)
                print remove_tags(list).encode('utf-8')     

execute(['scrapy','crawl','wiki'])

这将返回表中除目标数据之外的所有数据(它也不返回助攻，但我还没有为此添加任何逻辑.这段代码是我拥有的原始代码的开发既没有进球也没有助攻:

This returns all the data from the table apart from the goal data (it doesn't return assists either, but I haven't added any logic for that yet. This code is a development of the original piece of code I had which did not return goals or assists either:

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re


class MySpider(Spider):
    name = "goal"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]    

    def parse(self, response):
        titles = response.selector.xpath("normalize-space(//title)")
        for titles in titles:
            body = response.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]').extract()
            body2 = "".join(body)

            print remove_tags(body2).encode('utf-8')

execute(['scrapy','crawl','goal'])

HTML 源代码中指示目标的语句是这样的:

The statement in the HTML source that indicates a goal is this:

<span class="incident-wrapper"><span class="incidents-icon ui-icon goal" title="Goal"></span></span>

谁能告诉我为什么我在顶部列出的代码不返回以该逻辑得分的目标?这是否与球图标用于表示目标而不是单词的事实有关?

Can anyone tell me why my code listed at the top does not return goals scored with that logic? is it something to do with the fact that a ball icon is used to denote goals rather than a word?

谢谢

推荐答案

在第一个版本中你只会得到 <span class="incidents-icon ui-icon goal" title="Goal"></span> 并且没有文本，因此您会得到空字符串 - 因为您 remove_tags().

In first version you get only <span class="incidents-icon ui-icon goal" title="Goal"></span> and there is no text so you get empty string as result - because you remove_tags().

为带有目标图标"的行添加字符串GOAL":

Adding string "GOAL" for rows with "goal icon":

list_of_goals = row.xpath('//span[@title="Goal"]')

if list_of_goals:        
    list = str(list_of_goals)
    print remove_tags(list).encode('utf-8') + "GOAL" # <-- string

或(因为 中没有文字)

or (becasue there is no text in <span title="Goal"> )

list_of_goals = row.xpath('//span[@title="Goal"]')

if list_of_goals:
    print "GOAL" # <-- string

<小时>

我使用 Scrapy 0.22.2 制作了我的版本.

I made my version with Scrapy 0.22.2.

可能我们使用了不同版本的 Scrapy，因为您的某些功能对我不起作用.

Probably we use differrent version of Scrapy because some your functions did't work for me.

我更喜欢 css 选择器而不是 xpath - 它们对我来说更简单.

I prefer css selectors then xpath - they are simpler for me.

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.cmdline import execute


class MySpider(Spider):

    name = "goal"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]    

    def parse(self, response):
        sel = Selector(response)

        #titles = sel.xpath("normalize-space(//title)")
        #print 'titles:', titles.extract()[0]
        print
        print 'titles:', "".join( sel.css("title::text").extract() ).strip()
        print

        #rows = sel.xpath('//table[@id="player-fixture"]//tbody//tr')
        rows = sel.css('table#player-fixture tbody tr')

        for row in rows:
            #print 'date:', row.xpath('.//td[@class="date"]/text()').extract()
            #print 'result:', row.xpath('.//td[@class="result"]/a/text()').extract()
            print 'date:', "".join( row.css('.date::text').extract() ).strip()
            print 'result:', "".join( row.css('.result a::text').extract() ).strip()
            print 'team_home:', "".join( row.css('.team.home a::text').extract() ).strip()
            print 'team_away:', "".join( row.css('.team.away a::text').extract() ).strip()
            print 'info:', "".join( row.css('.info::text').extract() ).strip(), "".join( row.css('.info::attr(title)').extract() ).strip()
            print 'rating:', "".join( row.css('.rating::text').extract() ).strip()
            print 'incidents:', ", ".join( row.css('.incidents-icon::attr(title)').extract() ).strip()
            print '-'*40

#execute(['scrapy','crawl','goal'])
execute(['scrapy','runspider','main.py'])

和部分结果

titles: Wayne Rooney Match History | WhoScored.com

date: 17-08-2013
result: 1 : 4
team_home: Swansea
team_away: Manchester United
info: 28' Minutes played in this match
rating: 7.26
incidents: Assist, Assist
----------------------------------------
date: 26-08-2013
result: 0 : 0
team_home: Manchester United
team_away: Chelsea
info: 90' Minutes played in this match
rating: 7.03
incidents: 
----------------------------------------
date: 14-09-2013
result: 2 : 0
team_home: Manchester United
team_away: Crystal Palace
info: 90' Minutes played in this match
rating: 8.44
incidents: Man of the Match, Goal
----------------------------------------
date: 17-09-2013
result: 4 : 2
team_home: Manchester United
team_away: Bayer Leverkusen
info: 84' Minutes played in this match
rating: 9.18
incidents: Goal, Goal, Assist
----------------------------------------
date: 22-09-2013
result: 4 : 1
team_home: Manchester City
team_away: Manchester United
info: 90' Minutes played in this match
rating: 7.17
incidents: Goal, Yellow Card
----------------------------------------
date: 25-09-2013
result: 1 : 0
team_home: Manchester United
team_away: Liverpool
info: 90' Minutes played in this match
rating: 
incidents: Man of the Match, Assist
----------------------------------------

这篇关于Xpath() 方法不使用 Scrapy 返回结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！