如何使用不变的URL刮取多个页面 - Python 3 [英] How to scrape multiple pages with an unchanging URL - Python 3

问题描述

我最近接触过网页抓取并尝试网页抓取各种页面。目前，我正试图抓住以下网站 -

完成后，单击下一页按钮。您将看到以下文件。

单击该文件。在常规块中，您会看到我们需要的这两件事。

向下滚动，在表单数据标签中，您可以看到3个变量为

在这里，您可以看到更改 pageIndex 将提供所需的所有页面。

现在，我们已经获得了所有必需的数据，我们可以写一个 POST 使用上述数据的URL http://www.pizzahut.com.cn/StoreList/Index 的方法。

代码：

我会告诉你刮掉前两页的代码，你可以通过changin刮掉你想要的任意数量的页面g range（）。

  for page_no in range（1， 3）：
 data = {
'pageIndex'：page_no，
'pageSize'：10，
'keyword'：'输入餐厅地址或餐厅名称'
 } 
 page = requests.post（'http://www.pizzahut.com.cn/StoreList/Index'，data = data）
 soup = BeautifulSoup（page.text，'html.parser' ）
 
 print（'PAGE'，page_no）
 for row.find_all（'div'，class _ ='re_RNew'）中的行：
 name = row.find（' p'，class _ ='re_NameNew'）。string 
 info = row.find（'input'）。get（'value'）
 location = info.split（'|'）
 location_data = location [0] .split（'，'）
经度= location_data [0] 
纬度= location_data [1] 
打印（经度，纬度）
  

输出：

 第1页
 31.085877 121.399176 
 31.271117 121 .587577 
 31.098122 121.413396 
 31.331458 121.440183 
 31.094581 121.503654 
 31.270737000 121.481178000 
 31.138214 121.386943 
 30.915685 121.482079 
 31.279029 121.529255 
 31.168283 121.283322 
 PAGE 2 
 31.388674 121.35918 
 31.231706 121.472644 
 31.094857 121.219961 
 31.228564 121.516609 
 31.235717 121.478692 
 31.288498 121.521882 
 31.155139 121.428885 
 31.235249 121.474639 
 30.728829 121.341429 
 31.260372 121.343066 
  

---

注意：您可以通过更改 pageSize 的值来更改每页的结果（目前为10）。

I recently got in touch with web scraping and tried to web scrape various pages. For now, I am trying to scrape the following site - http://www.pizzahut.com.cn/StoreList

So far I've used selenium to get the longitude and latitude scraped. However, my code right now only extracts the first page. I know there is a dynamic web scraping that executes javascript and loads different pages, but had hard time trying to find a right solution. I was wondering if there's a way to access the other 49 pages or so, because when I click next page the URL does not change because it is set, so I cannot just iterate over a different URL each time

Following is my code so far:

import os
import requests
import csv
import sys
import time
from bs4 import BeautifulSoup

page = requests.get('http://www.pizzahut.com.cn/StoreList')

soup = BeautifulSoup(page.text, 'html.parser')

for row in soup.find_all('div',class_='re_RNew'):
    name = row.find('p',class_='re_NameNew').string
    info = row.find('input').get('value')
    location = info.split('|')
    location_data = location[0].split(',')
    longitude = location_data[0]
    latitude = location_data[1]
    print(longitude, latitude)

Thank you so much for helping out. Much appreciated

解决方案

Steps to get the data:

Open the developer tools in your browser (for Google Chrome it's Ctrl+Shift+I). Now, go to the XHR tab which is located inside the Network tab.

After doing that, click on the next page button. You'll see the following file.

Click on that file. In the General block, you'll see these 2 things that we need.

Scrolling down, in the Form Data tab, you can see the 3 variables as

Here, you can see that changing the value of pageIndex will give all the pages required.

Now, that we've got all the required data, we can write a POST method for the URL http://www.pizzahut.com.cn/StoreList/Index using the above data.

Code:

I'll show you the code to scrape first 2 pages, you can scrape any number of pages you want by changing the range().

for page_no in range(1, 3):
    data = {
        'pageIndex': page_no,
        'pageSize': 10,
        'keyword': '输入餐厅地址或餐厅名称'
    }
    page = requests.post('http://www.pizzahut.com.cn/StoreList/Index', data=data)
    soup = BeautifulSoup(page.text, 'html.parser')

    print('PAGE', page_no)
    for row in soup.find_all('div',class_='re_RNew'):
        name = row.find('p',class_='re_NameNew').string
        info = row.find('input').get('value')
        location = info.split('|')
        location_data = location[0].split(',')
        longitude = location_data[0]
        latitude = location_data[1]
        print(longitude, latitude)

Output:

PAGE 1
31.085877 121.399176
31.271117 121.587577
31.098122 121.413396
31.331458 121.440183
31.094581 121.503654
31.270737000 121.481178000
31.138214 121.386943
30.915685 121.482079
31.279029 121.529255
31.168283 121.283322
PAGE 2
31.388674 121.35918
31.231706 121.472644
31.094857 121.219961
31.228564 121.516609
31.235717 121.478692
31.288498 121.521882
31.155139 121.428885
31.235249 121.474639
30.728829 121.341429
31.260372 121.343066

---

Note: You can change the results per page by changing the value of pageSize (currently it's 10).

这篇关于如何使用不变的URL刮取多个页面 - Python 3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！