使用 BeautifulSoup 在评论标签中抓取表格 [英] Using BeautifulSoup to scrape tables within comment tags

问题描述

我正在尝试使用 BeautifulSoup 从以下网页中抓取表格:https://www.pro-football-reference.com/boxscores/201702050atl.htm

I am trying to scrape tables from the following webpage using BeautifulSoup: https://www.pro-football-reference.com/boxscores/201702050atl.htm

import requests
from bs4 import BeautifulSoup

url = 'https://www.pro-football-
reference.com/boxscores/201702050atl.htm'
page = requests.get(url)
html = page.text

页面上的大部分表格都在注释标签内，因此无法直接访问.

Most of the tables on the page are inside comment tags, so can't be accessed in a straightforward way.

print(soup.table.text)

返回:

1
2
3
4
OT
Final







via Sports Logos.net
About logos


New England Patriots
0
3
6
19 
6
34





via Sports Logos.net
About logos


Atlanta Falcons
0
21
7
0
0
28

即缺少包含玩家统计数据的主要表格.我试图简单地使用

i.e. the main tables containing the player stats are missing. I have tried to simply remove the comment tags using

html = html.replace('<!--',"")
html = html.replace('-->',"")

但无济于事.如何访问这些注释掉的表?

but to no avail. How can I access these commented-out tables?

推荐答案

给你.您可以从该页面获取任何表格，只需更改索引号即可.

Here you go. You can get any table from that page only changing the index number.

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.pro-football-reference.com/boxscores/201702050atl.htm').text

soup = BeautifulSoup(page,'lxml')
table = soup.find_all('table')[1]  #This is the index of any table of that page. If you change it you can get different tables.
tab_data = [[celldata.text for celldata in rowdata.find_all(["th","td"])]
                        for rowdata in table.find_all("tr")]
for data in tab_data:
    print(' '.join(data))

因为除了前两个表之外的其他表都在 javascript 中，这就是为什么您需要使用 selenium 来处理崩溃并解析它们.您现在绝对可以从该页面访问任何表格.这是修改后的.

As the other tables except for the first two are within javascript, that is why you need to use selenium to gatecrash and parse them. You will definitely be able to access any table from that page now. Here is the modified one.

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get('https://www.pro-football-reference.com/boxscores/201702050atl.htm')
soup = BeautifulSoup(driver.page_source,'lxml')
driver.quit()
table = soup.find_all('table')[7]  #This is the index of any table of that page. If you change it you can get different tables.
tab_data = [[celldata.text for celldata in rowdata.find_all(["th","td"])]
                        for rowdata in table.find_all("tr")]
for data in tab_data:
    print(' '.join(data))

这篇关于使用 BeautifulSoup 在评论标签中抓取表格的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！