使用python和xpath进行爬取 [英] Scraping using python and xpath

问题描述

我正在尝试从以下网站抓取数据:

I am trying to scrape data from the following website:

http://mozo.com.au/credit-cards/search#fetch/680

使用chrome的检查元素功能"，我可以将所需的元素地址定位为:

Using chrome's 'inspect element feature' I have been able to locate the element address I want as:

//*[@id="p-40"]/div[4]/table/tbody/tr/td[1]/text()

我希望使用此代码，我将能够获得文本"9.99％"

I was hoping using this code, I would be able to get the text "9.99%"

import requests
page = requests.get('http://mozo.com.au/credit-cards/search#fetch/680')
tree = html.fromstring(page.text)


tree.xpath('//*[@id="p-40"]/div[4]/table/tbody/tr/td[1]/text()')

但是，输出是一个空数组.我要去哪里错了?

However, the output is an empty array. Where am I going wrong?

推荐答案

就像 tobifasc 一样，该页面是动态加载的.以硒为例，

Like tobifasc said, the page is loaded dynamically. Try selenium for example,

首次安装:

pip3 install selenium

然后:

import lxml.html
from selenium import webdriver
driver = webdriver.Firefox()
driver.get(url)

tree = lxml.html.fromstring(driver.page_source)

现在您可以查询:

# With your xpath there are 2 results...
results = tree.xpath('//*[@id="p-40"]/div[4]/table/tbody/tr/td[1]/text()')   
results[1].strip()
'9.99%'

这篇关于使用python和xpath进行爬取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！