美丽的汤发现从rightmove不返回任何内容 [英] beautiful soup find returns none from rightmove

问题描述

我正尝试使用漂亮的汤从此处解析html:

和

有什么明显的我想念的东西吗?

解决方案

页面确实具有动态内容.而且，在抓取之前，您应该将selenium与网络驱动程序一起使用以加载所有内容.

您可以尝试在此处下载ChromeDriver可执行文件.如果将其粘贴到脚本所在的文件夹中，则可以运行:

  import os从硒导入webdriver从bs4导入BeautifulSoup#配置驱动程序chrome_options = webdriver.ChromeOptions()chrome_options.add_argument(-headless")chrome_driver = os.getcwd()+"\\ chromedriver.exe";#如果不在同一文件夹中，请更改此路径驱动程序= webdriver.Chrome(选项= chrome_options，可执行文件路径= chrome_driver)url ='https://www.rightmove.co.uk/house-prices/br5/broadcroft-road.html?page=1'driver.get(网址)page_soup =汤(driver.page_source，"html.parser")no_results = page_soup.find('div'，{'class':'section sort-bar-results'})容器= page_soup.findAll('div'，{'class':'propertyCard'})打印(no_results.text)print(len(containers)，"containers") 

您说的答案无济于事，但我在这里尝试过，结果表明:

  35个已售物业25个容器 

I am trying to use beautiful soup to parse the html from here: https://www.rightmove.co.uk/house-prices/br5/broadcroft-road.html?page=1

I have:

req=requests.get(url)
# page_soup = soup(req.content,'html.parser')
page_soup = soup(req.content,'lxml') 
no_results= page_soup.find('div',{'class':'section sort-bar-results'})
containers = page_soup.findAll('div',{'class':'propertyCard'})
no_results, len(containers)

this returns (None, 0)

I looked at Beautiful Soup find() returns None?, Beautiful Soup returns 'none', Beautiful soup returns None, Beautiful Soup returns None on existing element, but unfortunately none have helped me

The sections of the html correspond to:

and

Is there something obvious that I am missing?

解决方案

The page indeed have a dynamic content. And you should use selenium with a webdriver to load all the content before scraping.

You can try downloading ChromeDriver executable here. And if you paste it in the same folder as your script you can run:

import os
from selenium import webdriver
from bs4 import BeautifulSoup

# configure driver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_driver = os.getcwd() + "\\chromedriver.exe"  # IF NOT IN SAME FOLDER CHANGE THIS PATH
driver = webdriver.Chrome(options=chrome_options, executable_path=chrome_driver)

url = 'https://www.rightmove.co.uk/house-prices/br5/broadcroft-road.html?page=1'

driver.get(url)
page_soup = soup(driver.page_source, "html.parser")
no_results= page_soup.find('div',{'class':'section sort-bar-results'})
containers = page_soup.findAll('div',{'class':'propertyCard'})
print(no_results.text)
print(len(containers), "containers")

You said the answers didn't help but I tried it here and it ouputs:

35 sold properties
25 containers

这篇关于美丽的汤发现从rightmove不返回任何内容的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！