https://www.google.com/#q=vigilante+mic'中的soup.select('.r a')在python BeautifulSoup中给出了空列表 [英] soup.select('.r a') in 'https://www.google.com/#q=vigilante+mic' gives empty list in python BeautifulSoup

问题描述

我正在使用BeautifulSoup从Google搜索结果页面中提取所有链接.这是代码段:

I am using BeautifulSoup to extract all links from google search results page. here's the snippet of the code:

    import requests,bs4

    res = requests.get('https://www.google.com/#q=vigilante+mic')

    soup = bs4.BeautifulSoup(res.text)

    linkElem = soup.select('.r a')

但是soup.select('.r a')返回一个空列表

But soup.select('.r a') is returning an empty list

谢谢

推荐答案

那是因为您使用的是url:

That's because of the url you are using:

https://www.google.com/#q=vigilante+mic

是搜索的javascript版本.如果您 curl ，它将在html中看到没有答案.发生这种情况是因为结果是通过javascript获取的，而请求无法处理该结果.

Is a javascript version of the search. If you curl it you will see there are no answers in the html. This happens because the results are fetched through javascript and requests doesn't handle that.

尝试使用其他网址(不是基于javascript的网址):

Try this other url (that is not javascript based):

https://www.google.com/search?q=vigilante+mic

现在可以使用了:

import requests,bs4

res = requests.get('https://www.google.com/search?q=vigilante+mic')

soup = bs4.BeautifulSoup(res.text)

linkElem = soup.select('.r a')

这篇关于https://www.google.com/#q=vigilante+mic'中的soup.select('.r a')在python BeautifulSoup中给出了空列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！