如何使用 Beautifulsoup 解析网站 [英] How to parse the website using Beautifulsoup

问题描述

我是网络抓取的新手，我想获取页面的 html.但是当我运行程序时，我得到 html 为空，控制台显示 javascript

I am new to web scraping and i want to get the html of the page.But when i run the program i get html empty and console show the javascript

from bs4 import BeautifulSoup
import requests
import urllib

url = "https://linkedin.com/company/1005"

r = requests.get(url)
html_content = r.text
soup = BeautifulSoup(html_content,'html.parser')
print (soup.prettify())

推荐答案

问题不是 BeautifulSoup 而是服务器，它在请求中需要更多信息才能让您访问此页面.现在它会发送 JavaScript 代码，将您重定向到登录页面.

Problem is not BeautifulSoup but server which needs more information in requests to give you access to this page. Now it sends JavaScript code which redirects you to login page.

您需要 User-Agent 标头来获取此页面.

You need User-Agent header to get this page.

您可以使用http://httpbin.org/get查看User-Agent在您的浏览器中.

You can use http://httpbin.org/get to see User-Agent in your browser.

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0'}

url = "https://linkedin.com/company/1005"

r = requests.get(url, headers=headers)
print(r.text)

soup = BeautifulSoup(r.text, 'html.parser')
print(soup.prettify())

这篇关于如何使用 Beautifulsoup 解析网站的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！