当元素完全存在时，BeautifulSoup不返回任何元素 [英] BeautifulSoup returning none when element definitely exists

问题描述

我刚刚接触网络诈骗，并一直在使用BeautifulSoup来刮取每日抵押贷款利率。但是，我试图抓取的许多服务商网站都返回无或空列表。我试图刮去的标签当然存在，我只是不明白为什么会发生这种情况。一个例子如下。

  html = urlopen（'https://www.popular.com/en/mortgages/'）
 bs0bj = BeautifulSoup（html，'html.parser'）; 
 rate = bs0bj.find（'span'，{'class'：'text-md text-popular-medium-blue'}）。div 
 print（rate）
  
 
 
 ：
 
 
  from bs4导入BeautifulSoup $ b $ se from selenium import webdriver 
 
 driver = webdriver.Chrome （）
 driver.get（'https://www.popular.com/en/mortgages/'）
 
汤= BeautifulSoup（driver.page_source，lxml）
 item = soup.select（'。table-responsive'）[0] .select（span div）[0] .text 
 print（item）
 driver.quit（）
  
结果：
 
 
  2.75％
  
 
I'm new to web scraping and have been using BeautifulSoup to scrape daily mortgage rates. However, a lot of the servicer sites that I am trying to scrape return 'none' or an empty list. The tags that I'm trying to scrape definitely exist and I just don't understand why this is happening. One example is below. 
html=urlopen('https://www.popular.com/en/mortgages/')
bs0bj=BeautifulSoup(html, 'html.parser');
rate=bs0bj.find('span', {'class':'text-md text-popular-medium-blue'}).div
print(rate)

解决方案
To get the data you are after you can use selenium in combination with python something like below:
from bs4 import BeautifulSoup
from selenium  import webdriver

driver = webdriver.Chrome()
driver.get('https://www.popular.com/en/mortgages/')

soup = BeautifulSoup(driver.page_source,"lxml")
item = soup.select('.table-responsive')[0].select("span div")[0].text
print(item)
driver.quit()
Result:
2.75%


                        
这篇关于当元素完全存在时，BeautifulSoup不返回任何元素的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！