从Python这样的动态Web数据库的数据刮痧 [英] Scraping data from a dynamic web database with Python

问题描述

我是新来的Python和我目前正在设法弄清楚如何从这个网站抽取数据：

I am new to Python and am currently trying to figure out how to scrape data from this web:

的https://www.entsoe.eu/db-query/consumption/mhlv-a-specific-country-for-a-specific-month

我不知道如果我使用Scrapy，BeautifulSoup或硒。需要对特定的国家数据（DE说 - 德国）每个月和日内2012-2014

I am not sure if I use Scrapy, BeautifulSoup or Selenium. Need data for a specific country (say DE - Germany) for each month and day within 2012-2014.

任何帮助非常多AP preciated。

Any help is very much appreciated.

推荐答案

您可以用解决要求 （对于维护Web刮会议）+的 BeautifulSoup （对于HTML解析）+正则表达式包含内所需的数据中提取一个javascript变量的值剧本标签和 ast.literal_eval（） 制作一个Python列表出来JS名单：

You can solve it with requests (for maintaining a web-scraping session) + BeautifulSoup (for HTML parsing) + regex for extracting a value of a javascript variable containing the desired data inside a script tag and ast.literal_eval() for making a python list out of js list:

from ast import literal_eval
import re

from bs4 import BeautifulSoup
import requests


url = "https://www.entsoe.eu/db-query/consumption/mhlv-a-specific-country-for-a-specific-month"
payload = {
    'opt_period': '0',
    'opt_Country': '12',  # 12 stands for DE here
    'opt_Month': '1',
    'opt_Year': '2014',
    'opt_Response': '1',
    'send': 'send',
    'opt_period': '0'
}
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36'}

with requests.Session() as session:
    session.headers = headers
    session.get(url)

    response = session.post(url, data=payload)

    soup = BeautifulSoup(response.content)
    script = soup.find('script', text=re.compile(r'Ext.onReady')).text

    data = literal_eval(re.search(r"var myData = (.*?);", script, re.MULTILINE).group(1))
    for row in data:
        print row

打印：

['DE', '2014-01-01', 45424, 43537, 41773, 40716, 39945, 39014, 37282, 37573, 38225, 40639, 42884, 45332, 46285, 45671, 45293, 45840, 48863, 53721, 54607, 53691, 51219, 49701, 49099, 45850]
['DE', '2014-01-02', 42468, 40217, 39564, 39758, 41054, 43586, 48705, 54691, 58650, 61110, 62773, 64309, 64561, 63807, 62706, 61919, 63338, 66760, 66615, 64653, 60690, 57825, 55697, 51490]
['DE', '2014-01-03', 47538, 45125, 44358, 44748, 45815, 48024, 52151, 57564, 60767, 62425, 63654, 65152, 65273, 63591, 62195, 61722, 63311, 66785, 66668, 64317, 60460, 57727, 56084, 52332]
...
['DE', '2014-01-29', 57605, 55275, 54154, 54226, 55320, 58459, 66647, 73890, 75957, 75958, 76725, 77446, 76852, 76362, 75300, 74549, 73958, 77129, 78240, 76323, 71961, 68595, 66088, 61923]
['DE', '2014-01-30', 58207, 56235, 54953, 54873, 55861, 58952, 66756, 73747, 75479, 75507, 76249, 76763, 76013, 75291, 73975, 73267, 72717, 76181, 77765, 76038, 71807, 68369, 65580, 61414]
['DE', '2014-01-31', 57870, 55665, 54381, 54422, 55419, 58490, 65929, 72706, 74666, 74392, 74791, 74923, 73877, 72205, 70449, 69596, 69345, 73259, 74950, 72959, 68623, 65319, 63414, 59467]

具体硒的方法是不那么神奇，但我认为这是绰绰有余你开始（并以最小的研究努力的问题）。

Selenium-specific approach would be less "magical", but I think this is more than enough for you to start (and for a question with minimal research effort).

这篇关于从Python这样的动态Web数据库的数据刮痧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！