为什么在我的网页抓取程序中没有解析任何内容? [英] why is nothing getting parsed in my web scraping program?

问题描述

我制作了这段代码来搜索谷歌搜索中的所有顶级链接.但它没有返回.

导入浏览器，请求从 bs4 导入 BeautifulSoup字符串 = '赛琳娜+戈麦斯'网站 = f'http://google.com/search?q={string}'req_web = requests.get(website).text解析器 = BeautifulSoup(req_web, 'html.parser')gotolink = parser.find('div', class_='r').a["href"]打印(转到链接)

解决方案

Google 需要您指定 User-Agent http 标头以返回正确的页面.如果没有指定正确的 User-Agent，Google 将返回不包含

标签和 r 类的页面.当你在有和没有 User-Agent 的情况下执行 print(soup) 时，你可以看到它.

例如:

导入请求从 bs4 导入 BeautifulSoup字符串 = '赛琳娜+戈麦斯'headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0'}网站 = f'http://google.com/search?hl=en&q={string}'req_web = requests.get(website, headers=headers).text解析器 = BeautifulSoup(req_web, 'html.parser')gotolink = parser.find('div', class_='r').a["href"]打印(转到链接)

打印:

https://www.instagram.com/selenagomez/?hl=en

I made this code to search all the top links in google search. But its returning none.

import webbrowser, requests
from bs4 import BeautifulSoup
string = 'selena+gomez'
website = f'http://google.com/search?q={string}'
req_web = requests.get(website).text
parser = BeautifulSoup(req_web, 'html.parser')
gotolink = parser.find('div', class_='r').a["href"]
print(gotolink)

解决方案

Google needs that you specify User-Agent http header to return correct page. Without the correct User-Agent specified, Google returns page that doesn't contain <div> tags with r class. You can see it when you do print(soup) with and without User-Agent.

For example:

import requests
from bs4 import BeautifulSoup

string = 'selena+gomez'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0'}
website = f'http://google.com/search?hl=en&q={string}'

req_web = requests.get(website, headers=headers).text
parser = BeautifulSoup(req_web, 'html.parser')
gotolink = parser.find('div', class_='r').a["href"]
print(gotolink)

Prints:

https://www.instagram.com/selenagomez/?hl=en

这篇关于为什么在我的网页抓取程序中没有解析任何内容?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！