如何使用Beautifulsoup解析网站 [英] How to parse the website using Beautifulsoup
本文介绍了如何使用Beautifulsoup解析网站的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我是Web抓取的新手,我想获取页面的html.但是,当我运行该程序时,我得到的html为空,控制台显示了javascript
I am new to web scraping and i want to get the html of the page.But when i run the program i get html empty and console show the javascript
from bs4 import BeautifulSoup
import requests
import urllib
url = "https://linkedin.com/company/1005"
r = requests.get(url)
html_content = r.text
soup = BeautifulSoup(html_content,'html.parser')
print (soup.prettify())
推荐答案
问题不是BeautifulSoup
,而是服务器,它需要请求中提供更多信息以使您能够访问此页面.现在,它发送JavaScript代码,将您重定向到登录页面.
Problem is not BeautifulSoup
but server which need more information in requests to give you access to this page. Now it sends JavaScript code which redirects you to login page.
您需要User-Agent
标头才能获得此页面.
You need User-Agent
header to get this page.
您可以使用 http://httpbin.org/get 在浏览器中查看User-Agent
.
You can use http://httpbin.org/get to see User-Agent
in your browser.
import requests
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Mozilla/5.0'}
url = "https://linkedin.com/company/1005"
r = requests.get(url, headers=headers)
print(r.text)
soup = BeautifulSoup(r.text, 'html.parser')
print(soup.prettify())
这篇关于如何使用Beautifulsoup解析网站的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文