与网络请求刮板分页 [英] Paginate with network requests scraper

问题描述

我正在尝试刮除Naukri的职位空缺.Web抓取非常耗时，因此我切换到了网络请求.我相信我可以通过正确更改URL(而不是单击下一个选项卡)来获得分页的请求模式.

I am trying to scrape Naukri job postings. Web scraping was too time-consuming, so I switched to network requests. I believe I got the request pattern for pagination by changing the URL right (not clicking the next tab).

URL示例:

https://www.naukri.com/maintenance-jobs?xt = catsrch& qf％5B％5D = 19

https://www.naukri.com/maintenance-jobs-2?xt = catsrch& qf％5B％5D = 19

https://www.naukri.com/maintenance-jobs-3?xt = catsrch& qf％5B％5D = 19

https://www.naukri.com/maintenance-jobs-4?xt = catsrch& qf％5B％5D = 19

以下代码可在首页上正常运行.但是，它无法分页并从下一个请求链接获取数据.相反，对于每次运行，它都是从第一个链接获取数据.

The below code works on the first page properly. However, it is failing to paginate and get data from the next request link. Instead, for each run, it is getting data from the first link.

我要去哪里了?如何正确地对URL请求进行分页?

import pandas as pd
import time
import re
from bs4 import BeautifulSoup
import unicodecsv
import requests
import ast
import json
import time
import logging
import datetime

headers_naukri={
    "authority": 'www.naukri.com',
    "user-agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.70',
    'x-newrelic-id': 'VgUHWVFXABACXVdaBAgHUF0=',
    "systemid": '109',
    "content-type": "application/json",
    "accept": 'application/json',
    "cache-control": "no-cache",
    "clientid": 'd3skt0p',
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    'referer': 'https://www.naukri.com/maintenance-jobs?xt=catsrch&qf[]=19',
    "sec-fetch-site": "same-origin",
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "en-US,en;q=0.9",
    "appid": '109',
    "content-type": "application/json"
}

session = requests.Session()
source_url='https://www.naukri.com/jobapi/v3/search?noOfResults=20&urlType=search_by_keyword&searchType=adv&keyword=maintenance&xt=catsrch&functionAreaId=19&seoKey=maintenance-jobs&src=jobsearchDesk&latLong='
print(source_url)
try:
    req = session.get(source_url, headers = headers_naukri)
except requests.exceptions.TooManyRedirects:
    time.sleep(15)
    req = session.get(source_url,headers=headers_naukri)
source_name = 'naukri'
source_id = '1'
source = req.text
#print(source)
source_json = json.loads(source)
noOfJobs=source_json['noOfJobs']
print(noOfJobs)
pages = (noOfJobs // 20) + 1
print(pages)
print(source_json['jobDetails'][0]['jobId'])
a=headers_naukri['referer'].split("?")
for i in range(2,pages,1):
    nexturl=source_url.replace('&src=', '-'+str(i)+'&src=')
    nexturl=nexturl.replace('&xt', '&pageNo='+str(i)+'&xt')
    headers_naukri['referer']= a[0]+'-'+str(i)+'?'+a[1]
    print(headers_naukri['referer'])
    #print(nexturl)
    try:
        req = session.get(source_url, headers = headers_naukri)
        headers_naukri['referer']= headers_naukri['referer'][:-18]+'-'+str(i)+headers_naukri['referer'][-18:]
    except requests.exceptions.TooManyRedirects:
        time.sleep(15)
        req = session.get(source_url,headers=headers_naukri)
    source_name = 'naukri'
    source_id = '1'
    source = req.text
    #print(source)
    source_json = json.loads(source)
    print(source_json['jobDetails'][1]['jobId'])

输出:

https://www.naukri.com/jobapi/v3/search?noOfResults=20&urlType=search_by_keyword&searchType=adv&keyword=maintenance&xt=catsrch&functionAreaId=19&seoKey=maintenance-jobs&src=jobsearchDesk&latLong=
143263
7164
300920001858
https://www.naukri.com/maintenance-jobs-2?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-3?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-4?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-5?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-6?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-7?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-8?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-9?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-10?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-11?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-12?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-13?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-14?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-15?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-16?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-17?xt=catsrch&qf[]=19

您可以清楚地看到，在每个请求中，第一个职位信息是相同的，并且没有分页.

You can clearly see that on each request, the first job post id is the same, and it is not paginating.

推荐答案

只需通过更新有效内容中的 pageNo 值并从 JSON 中获取所需内容，即可遍历页面.代码>.

Just loop through the pages by updating the pageNo value in the payload and get whatever you need from the JSON.

下面是一个如何获取前三页工作结果的示例:

Here's an example of how to get job results for the first three pages:

from urllib.parse import urlencode

import requests


url = "https://www.naukri.com/jobapi/v3/search?"


headers = {
    "appid": "109",
    "systemid": "109",
    "clientid": "d3skt0p",
    "accept": "application/json",
    "accept-encoding": "gzip, deflate, br",
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36",
}

payload = {
    "noOfResults": 20,
    "urlType": "search_by_keyword",
    "searchType": "adv",
    "keyword": "maintenance",
    "xt": "catsrch",
    "functionAreaId": 19,
    "seoKey": "maintenance-jobs-3",
    "src": "jobsearchDesk",
}

for page_number in range(1, 4):
    payload['pageNo'] = page_number
    search_results = requests.get(url, params=urlencode(payload),  headers=headers).json()
    for job in search_results['jobDetails']:
        print(job['title'])

输出:

Assistant Manager - Maintenance and Engineering
Maintenance Engineer
Electrical Maintenance Engineer - DRI Plant - Nigeria Location
Shift Engineer Mechanical Maintenance - DRI Plant - Nigeria Location
Maintenance Engineer I
Maintenance Engineer I
Senior Engineer/ Assistant Manager - Textile Maintenance
Senior Manager - Electrical - Project & Maintenance –WHRB-Nigeria
Core Maintenance Engg Utilities
State Core Maintenance Engg Utilities
State Core Maintenance Engg Fiber
State Core Maintenance Engg Utilities
State Core Maintenance Engg Utilities 
Core Maintenance Engg Utilities
Electrical Maintenance Engineer 
Electrical Engineer - Operations & Maintenance
Electrical Maintenance Engineer,
Job opening  For Jr  Maintenance Engineer  For Hospital in Pune
Assistant Manager Maintenance
Mech Maintenance Head - Sr. Manager - DRI Plant - Nigeria Location
Deputy Manager Maintenance - At Plant Level Manesar Sec 3
Senior Manager - Mechanical Maintenance – WHRB  (For Nigeria)
Sand Plant Mechanical Maintenance Engineer/ Sr. Engineer
Urgent Requirement For Maintenance Engineer For Rudrapur
CNC Service Engineer / Maintenance Engineer (Mechanical)
Operations & Maintenance, Mech Engr, Natural Gas Processing, Nigeria
Maintenance Engineer
CNC Maintenance Engineer
Utility Maintenance Engineer
Engineer - Maintenance
Machine Maintenance Engineer
Maintenance Engineer
Maintenance Engineer
Maintenance Engineer - Textile Industry
MACHINERY MAINTENANCE ENGINEER
Engineer - Maintenance
Assistant Engineer - Maintenance
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Site Engineer/ Sr. Engineer- Operation & Maintenance
Senior Aircraft Maintenance Engineering
Assistant Engineer - Maintenance
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Site Engineer/ Sr. Engineer- Operation & Maintenance
Electrical Maintenance - Telangana
Engineer/Sr. Engineer/AM in Production, Quality, Process, 
Maintenance
Jr. Maintenance Engineer (Utility) 
Operation & Maintenance Engineer
Electrical Maintenance Technician-sri City
Electrical Maintenance Technician
Maintenance Manager
Maintenance Manager
Sr. Exe. / Asst. Mgr. - Plant Maintenance & Engineering Deptt.- Dubai
Senior Engineering(Maintenance) Manager
Team Member Maintenance
Fleet Executive (service/repair/maintenance) Anywhere in U.P.
Executive- Maintenance(Sterile API)
Mechanical Maintenance Technicians - Sodexo Sriperumbudur location
Mechanical Maintenance Technicians - Sodexo Sriperumbudur location

这篇关于与网络请求刮板分页的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！