使用 BeautifulSoup 抓取 Web 数据 [英] Scraping Web data using BeautifulSoup
本文介绍了使用 BeautifulSoup 抓取 Web 数据的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试从 rotowire.com 获取每场棒球比赛的降雨机会和温度/风速.抓取数据后,我会将其转换为三列 - 雨、温度和风.感谢另一位用户,我能够接近获取数据,但无法完全到达那里.我尝试了两种方法.
I am trying to scrape the rain chance and the temperature/wind speed for each baseball game from rotowire.com. Once I scrape the data, I will then convert it to three columns - rain, temperature, and wind. Thanks to another user, I was able to get close to getting the data but cannot quite get all the way there. I've tried two approaches.
第一种方法:
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://www.rotowire.com/baseball/daily-lineups.php'
r = requests.get(url)
soup = BeautifulSoup(r.text, "html.parser")
weather = []
for i in soup.select(".lineup__bottom"):
forecast = i.select_one('.lineup__weather-text').text
weather.append(forecast)
返回:
['\n100% Rain\r\n 66°\xa0\xa0Wind 8 mph In ', '\n0% Rain\r\n 64°\xa0\xa0Wind 4 mph L-R ', '\n0% Rain\r\n 69°\xa0\xa0Wind 7 mph In ', '\nDome\r\n In Domed Stadium\r\n ', '\n0% Rain\r\n 75°\xa0\xa0Wind 10 mph Out ', '\n0% Rain\r\n 68°\xa0\xa0Wind 9 mph R-L ', '\n0% Rain\r\n 82°\xa0\xa0Wind 9 mph ', '\n0% Rain\r\n 81°\xa0\xa0Wind 5 mph R-L ', '\nDome\r\n In Domed Stadium\r\n ', '\n1% Rain\r\n 75°\xa0\xa0Wind 4 mph R-L ', '\n1% Rain\r\n 71°\xa0\xa0Wind 6 mph Out ', '\nDome\r\n In Domed Stadium\r\n ']
我尝试过的第二种方法是:
The second approach I've tried is:
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://www.rotowire.com/baseball/daily-lineups.php'
r = requests.get(url)
soup = BeautifulSoup(r.text, "html.parser")
#weather = []
for i in soup.select(".lineup__bottom"):
forecast = i.select_one('.lineup__weather-text').text
weather.append(forecast)
#print(forecast)
rain = i.select_one('.lineup__weather-text b:contains("Rain") ~ span').text
这将返回一个 AttributeError 'NoneType' 对象没有属性 'text'
推荐答案
要定位所有数据,请参见此示例:
To locate all the data, see this example:
import pandas as pd
import requests
from bs4 import BeautifulSoup
url = "https://www.rotowire.com/baseball/daily-lineups.php"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
weather = []
for tag in soup.select(".lineup__bottom"):
header = tag.find_previous(class_="lineup__teams").get_text(
strip=True, separator=" vs "
)
rain = tag.select_one(".lineup__weather-text > b")
forecast_info = rain.next_sibling.split()
temp = forecast_info[0]
wind = forecast_info[2]
weather.append(
{"Header": header, "Rain": rain.text.split()[0], "Temp": temp, "Wind": wind}
)
df = pd.DataFrame(weather)
print(df)
输出:
Header Rain Temp Wind
0 PHI vs CIN 100% 66° 8
1 CWS vs CLE 0% 64° 4
2 SD vs CHC 0% 69° 7
3 NYM vs ARI Dome In Stadium
4 MIN vs BAL 0% 75° 9
5 TB vs NYY 0% 68° 9
6 MIA vs TOR 0% 81° 6
7 WAS vs ATL 0% 81° 4
8 BOS vs HOU Dome In Stadium
9 TEX vs COL 0% 76° 6
10 STL vs LAD 0% 73° 4
11 OAK vs SEA Dome In Stadium
这篇关于使用 BeautifulSoup 抓取 Web 数据的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文