python selenium 自动加载更多页面 [英] python selenium automatically load more pages

问题描述

当我们到达页面底部时，我正在尝试为每个产品线创建一个网站www.jabong.com"，它会加载更多产品.我想报废所有链接.我正在尝试的代码如下所示:

I am trying a scrap a website, "www.jabong.com" here for each product line when we reach the bottom of the page it load more products. I want to scrap all the links. The code which I am trying is as shown below:

from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options


def fetch_links(url, product_line_name):

    chrome_options = Options()
    chrome_options.add_argument("--disable-notifications")
    chrome_path = r"D:\chromedriver.exe"
    driver = webdriver.Chrome(chrome_path, chrome_options=chrome_options)
    driver.get(url)
    button="load-more-products"
    while True:
        element=driver.find_element_by_class_name(button).click()
        driver.execute_script("arguments[0].scrollIntoView();", element)



link_list=["https://www.jabong.com/women/clothing/trousers-jeans/trousers/?source=topnav_women"]
product_line=["trousers"]

fetch_links(link_list[0],product_line[0])

这里的问题是无法定位元素.我也尝试过使用 class_name、css_selector，但它们似乎都不起作用.我想自动在浏览器上连续加载产品.

The problem here is unable to locate the element. I have also tried with class_name, css_selector but non of them seems to work. I want to load the product continuously on browser automatically.

推荐答案

试试这个.它将获取您所需的数据.

Try this. It will fetch you required data.

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys

link_list = ["https://www.jabong.com/women/clothing/trousers-jeans/trousers/?source=topnav_women",]

def fetch_links(link):
    chrome_options = Options()
    chrome_options.add_argument("--disable-notifications")
    driver = webdriver.Chrome(chrome_options=chrome_options)
    driver.get(link)
    actions = ActionChains(driver)
    for _ in range(10):   #Adjust this range according to your need, I meant how far you wanna go down.
        actions.send_keys(Keys.SPACE).perform()
        time.sleep(1)

    for item in driver.find_elements_by_css_selector(".h4"):
        print(item.text)

fetch_links(link_list[0])

部分结果:

W Orange Printed Palazzo
DOROTHY PERKINS Olive Solid Mid Rise Skinny Fit Coloured Pant
VARANGA White Solid Palazzo
VARANGA Blue Printed Palazzo

这篇关于python selenium 自动加载更多页面的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！