无法使用beautifulsoup抓取Google搜索结果 [英] Cant scrape google search results with beautifulsoup

问题描述

我想抓取Google搜索结果，但是每当我尝试这样做时，程序都会返回一个空列表

I want to scrape google search results , but whenever i try to do so, the program returns an empty list

from bs4 import BeautifulSoup
import requests

keyWord = input("Input Your KeyWord :")

url = f'https://www.google.com/search?q={keyWord}'
src = requests.get(url).text
soup = BeautifulSoup(src, 'lxml')

container = soup.findAll('div', class_='g')

print(container)

推荐答案

补充 Andrej Kesely's 的答案，如果您得到空的结果您总是可以爬上一个 div 上或下进行测试并从那里进行.

Complementing Andrej Kesely's answer if you getting empty results you always can climb one div up or down to test and go from there.

代码(例如，您要抓取 title ， summary 和 link ):

Code (say you want to scrape title, summary and link):

from bs4 import BeautifulSoup
import requests
import json

headers = {
    'User-agent':
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

html = requests.get('https://www.google.com/search?q=ice cream',
                    headers=headers).text

soup = BeautifulSoup(html, 'lxml')

summary = []

for container in soup.findAll('div', class_='tF2Cxc'):
  heading = container.find('h3', class_='LC20lb DKV0Md').text
  article_summary = container.find('span', class_='aCOpRe').text
  link = container.find('a')['href']

  summary.append({
      'Heading': heading,
      'Article Summary': article_summary,
      'Link': link,
  })

print(json.dumps(summary, indent=2, ensure_ascii=False))

输出部分:

[
  {
    "Heading": "Ice cream - Wikipedia",
    "Article Summary": "Ice cream (derived from earlier iced cream or cream ice) is a sweetened frozen food typically eaten as a snack or dessert. It may be made from dairy milk or cream and is flavoured with a sweetener, either sugar or an alternative, and any spice, such as cocoa or vanilla.",
    "Link": "https://en.wikipedia.org/wiki/Ice_cream"
  },
  {
    "Heading": "Jeni's Splendid Ice Creams",
    "Article Summary": "Jeni's Splendid Ice Cream, built from the ground up with superlative ingredients. Order online, visit a scoop shop, or find the closest place to buy Jeni's near you.",
    "Link": "https://jenis.com/"
  }
]

---

或者，您可以使用SerpApi的 Google搜索引擎结果API 进行此操作.这是一个带有免费试用版的痛苦API.查看游乐场.

---

Alternatively, you can do this using Google Search Engine Results API from SerpApi. It's a pain API with a free trial. Check out the Playground.

import os
from serpapi import GoogleSearch

params = {
    "engine": "google",
    "q": "ice cream",
    "api_key": os.getenv("API_KEY"),
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results["organic_results"]:
  print(f"Title: {result['title']}\nSummary: {result['snippet']}\nLink: {result['link']}\n")

输出部分:

Title: Ice cream - Wikipedia
Summary: Ice cream (derived from earlier iced cream or cream ice) is a sweetened frozen food typically eaten as a snack or dessert. It may be made from dairy milk or cream and is flavoured with a sweetener, either sugar or an alternative, and any spice, such as cocoa or vanilla.
Link: https://en.wikipedia.org/wiki/Ice_cream

Title: 6 Ice Cream Shops to Try in Salem, Massachusetts ...
Summary: 6 Ice Cream Shops to Try in Salem, Massachusetts · Maria's Sweet Somethings, 26 Front Street · Kakawa Chocolate House, 173 Essex Street · Melt ...
Link: https://www.salem.org/icecream/

Title: Melt Ice Cream - Salem
Summary: Homemade ice cream made on-site in Salem, MA. Bold innovative flavors, exceptional customer service, local ingredients.
Link: https://meltsalem.com/

免责声明，我为SerpApi工作.

Disclaimer, I work for SerpApi.

这篇关于无法使用beautifulsoup抓取Google搜索结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！