使用Python中的可点击内容来抓取网站 [英] Scraping a website with clickable content in Python

问题描述

我想在以下网站上抓取内容:

I would like to scrap the content a the following website:

http://financials.morningstar.com/ratios/r.html? t = AMD

在其中 Key Ratios 下，我想单击"Growth"按钮，然后在Python中抓取数据.

In there under Key Ratios I would like to click on "Growth" button and then scrap the data in Python.

我该怎么做?

推荐答案

您可以使用requests + BeautifulSoup来解决.发送到 http://financials.morningstar.com/financials/的异步GET请求您需要模拟的getKeyStatPart.html 端点. Growth表位于div中，且带有id="tab-growth":

You can solve it with requests+BeautifulSoup. There is an asynchronous GET request sent to the http://financials.morningstar.com/financials/getKeyStatPart.html endpoint which you need to simulate. The Growth table is located inside the div with id="tab-growth":

from bs4 import BeautifulSoup
import requests


url = 'http://financials.morningstar.com/ratios/r.html?t=AMD'
keystat_url = 'http://financials.morningstar.com/financials/getKeyStatPart.html'

with requests.Session() as session:
    session.headers = {'User-Agent': 'Mozilla/5.0 (Linux; U; Android 4.0.3; ko-kr; LG-L160L Build/IML74K) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30'}

    # visit the target url
    session.get(url)

    params = {
        'callback': '',
        't': 'XNAS:AMD',
        'region': 'usa',
        'culture': 'en-US',
        'cur': '',
        'order': 'asc',
        '_': '1426047023943'
    }
    response = session.get(keystat_url, params=params)

    # get the HTML part from the JSON response
    soup = BeautifulSoup(response.json()['componentData'])

    # grab the data
    for row in soup.select('div#tab-growth table tr'):
        print row.text

这篇关于使用Python中的可点击内容来抓取网站的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！