使用xpath搜寻新的ESPN网站[Python] [英] Scraping new ESPN site using xpath [Python]

问题描述

我正在尝试刮擦新的ESPN NBA计分板.这是一个简单的脚本，该脚本应返回15年4月5日所有游戏的开始时间:

I am trying to scrape the new ESPN NBA scoreboard. Here is a simple script which should return the start times for all games on 4/5/15:

import requests
import lxml.html
from lxml.cssselect import CSSSelector

doc =  lxml.html.fromstring(requests.get('http://scores.espn.go.com/nba/scoreboard?date=20150405').text)

#xpath
print doc.xpath("//title/text()") #print page title
print doc.xpath("//span/@time")
print doc.xpath("//span[@class='time']")
print doc.xpath("//span[@class='time']/text()")

#CCS Selector
sel = CSSSelector('span.time')
for i in sel(doc):
    print i.text

它不返回任何内容，但返回页面标题:

It doesn't return anything, but the page title:

['NBA Basketball Scores - NBA Scoreboard - ESPN']
[]
[]
[]

任何人都可以帮忙吗?谢谢

Can anyone help? Thanks

推荐答案

页面的性质是非常动态的-存在异步XHR请求和所涉及的javascript逻辑. requests不是浏览器，并且仅下载初始HTML页面，并且在requests获取的HTML中没有span元素包含class="time".

The nature of the page is quite dynamic - there are asynchronous XHR requests, javascript logic involved. requests is not a browser and downloads only the initial HTML page and there are no span elements with class="time" in the HTML that requests gets.

解决该问题的一种方法是使用 selenium 的真实浏览器. .这是使用PhantomJS无头浏览器的示例:

One of the options to approach the problem would be to involve a real browser using selenium. Here is an example using PhantomJS headless browser:

>>> from selenium import webdriver
>>> 
>>> url = "http://scores.espn.go.com/nba/scoreboard?date=20150405"
>>> 
>>> driver = webdriver.PhantomJS()
>>> driver.get(url)
>>> 
>>> elements = driver.find_elements_by_css_selector("span.time")
>>> for element in elements:
...     print element.text
... 

1:00 PM ET
3:30 PM ET
6:00 PM ET
7:00 PM ET
7:30 PM ET
9:00 PM ET
9:30 PM ET 

---

或者，您可以使用id="scoreboard-page"在div的data-data属性中查找所需的数据:

---

Alternatively, you can look for the desired data in the data-data attribute of the div with id="scoreboard-page":

import json
from pprint import pprint

import lxml.html
import requests

response = requests.get('http://scores.espn.go.com/nba/scoreboard?date=20150405')
doc = lxml.html.fromstring(response.content)

data = doc.xpath("//div[@id='scoreboard-page']/@data-data")[0]
data = json.loads(data)

pprint(data)

这篇关于使用xpath搜寻新的ESPN网站[Python]的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！