在lxml中处理分页 [英] Handling pagination in lxml

问题描述

我试图镜像一个我写的ruby scraper，但是只用于python环境。我决定使用lxml和请求来完成这个任务。我的问题是分页：

I am trying to mirror a ruby scraper that I wrote but for a python only environment. I've decided to use lxml and requests to get this done. My problem is pagination:

base_url = "http://example.com/something/?page=%s"  
for url in [base_url % i for i in xrange(10)]:  
    r = requests.get(url)

我是python和这个库的新手，所以我不确定执行等效ruby代码的最佳方式：

I'm new to python and this library so I'm not sure the best way to perform the equivalent ruby code:

last_pg = (page.xpath("//div[contains(@class, 'b-tabs-utility')]").text.split('of ')[-1].split(' Results')[0].to_f / 60).ceil
puts "Current Index: #{last_pg}"
for pg_number in 1..last_pg do
  puts "Getting links on page #{pg_number}"

推荐答案

获取结果数量通过将 b-tabs-utility div拆分为空格并获取最后一个元素：

Get the number of results by splitting the b-tabs-utility div into spaces and getting element before last:

base_url = "http://example.com/something/?page=%d" 
results_per_page = 60

div = page.xpath("//div[contains(@class, 'b-tabs-utility')]")[0].text
last_pg = int(div.split()[-2]) / results_per_page
for i in xrange(last_pg):
    r = requests.get(base_url % i)

我是 div 文本采用以下格式，例如：在单页面上工作但不失误 ... 642个结果

I'm assuming the div text is in the following format, example: ... of 642 Results

这篇关于在lxml中处理分页的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！