如何仅使用Selenium Python中的一个类就可以从任何网站获取所有数据 [英] How to get all the data from any website with just one class in selenium python

问题描述

我想用硒刮掉整个网站.我在亚马逊上得到了一类产品名称.我只想将所有产品名称归为一个类名.无需为每个产品手动复制任何ID或XPATH.怎么做??

I want to scrape the whole website with selenium. I got one class of a product name in amazon. I just want to get all the product names under one class name. Without manually copying any id's or XPATH's for each and every product. How to do that??

我在justdial.com上曾尝试过的方法:

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) ' \
             'Chrome/80.0.3987.132 Safari/537.36'

driver_exe = 'chromedriver'
options = ChromeOptions()
options.add_argument("--headless")
options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome(options=options)

driver.get("https://www.justdial.com/Bangalore/Bakeries")
x = driver.find_elements_by_class_name("store-name")

for i in x:
    print(i.text)

我在amazon.com上尝试过的内容

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) ' \
             'Chrome/80.0.3987.132 Safari/537.36'

driver_exe = 'chromedriver'
options = Options()
options.add_argument("--headless")
options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome(executable_path=r"C:\Users\intel\Downloads\Setups\chromedriver.exe", options=options)
driver.get("https://www.amazon.com/s?k=asus&rh=n%3A300189&nav_sdd=aps&pd_rd_r=58b28d7d-1955-433b-b33b-b1b5dcf1f522&pd_rd_w=MJzan&pd_rd_wg=QG3cj&pf_rd_p=6d81377b-6d6c-4363-ae02-8fa202ed7b50&pf_rd_r=X0BDDAPN7TTW0ZT1REX6&qid=1583290662&ref=sxwds-sbc_c2")
x = driver.find_elements_by_class_name("a-size-medium a-color-base a-text-normal")

for i in list(x):
    print(i.text.strip())

正在显示带有亚马逊的输出:

"A cookie associated with a cross-site resource at http://yahoo.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.", source:
https://www.amazon.com/s?k=asus&rh=n%3A300189&nav_sdd=aps&pd_rd_r=58b28d7d-1955-433b-b33b-b1b5dcf1f522&pd_rd_w=MJzan&pd_rd_wg=QG3cj&pf_rd_p=6d81377b-6d6c-4363-ae02-8fa202ed7b50&pf_rd_r=X0BDDAPN7TTW0ZT1REX6&qid=1583290662&ref=sxwds-sbc_c2 (0)
[0306/071643.332:INFO:CONSOLE(0)] "A cookie associated with a cross-site resource at https://yahoo.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.", source: https://www.amazon.com/s?k=asus&rh=n%3A300189&nav_sdd=aps&pd_rd_r=58b28d7d-1955-433b-b33b-b1b5dcf1f522&pd_rd_w=MJzan&pd_rd_wg=QG3cj&pf_rd_p=6d81377b-6d6c-4363-ae02-8fa202ed7b50&pf_rd_r=X0BDDAPN7TTW0ZT1REX6&qid=1583290662&ref=sxwds-sbc_c2 (0)
[0306/071644.820:INFO:CONSOLE(0)] "A cookie associated with a cross-site resource at https://surveywall-api.survata.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.", source: https://www.amazon.com/s?k=asus&rh=n%3A300189&nav_sdd=aps&pd_rd_r=58b28d7d-1955-433b-b33b-b1b5dcf1f522&pd_rd_w=MJzan&pd_rd_wg=QG3cj&pf_rd_p=6d81377b-6d6c-4363-ae02-8fa202ed7b50&pf_rd_r=X0BDDAPN7TTW0ZT1REX6&qid=1583290662&ref=sxwds-sbc_c2 (0)

我只想要一个可以在任何/大多数站点中执行的硒脚本...任何帮助，将不胜感激.

I just want one selenium script which will perform with any/most of the sites... Any help would be appreciated.

推荐答案

.find_elements_by_class_name仅用于单个类名，如果有多个类名，则可以使用.find_elements_by_css_selector.

.find_elements_by_class_name just for single class name, if multiple class name you can use .find_elements_by_css_selector.

尝试一下:

x = driver.find_elements_by_css_selector(".a-size-medium.a-color-base.a-text-normal")

这篇关于如何仅使用Selenium Python中的一个类就可以从任何网站获取所有数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！