使用BeautifulSoup在评论标签内抓取表格 [英] Using BeautifulSoup to scrape tables within comment tags

问题描述

我正在尝试使用BeautifulSoup从以下网页中抓取表格: https://www.pro-football-reference.com/boxscores/201702050atl. htm

I am trying to scrape tables from the following webpage using BeautifulSoup: https://www.pro-football-reference.com/boxscores/201702050atl.htm

import requests
from bs4 import BeautifulSoup

url = 'https://www.pro-football-
reference.com/boxscores/201702050atl.htm'
page = requests.get(url)
html = page.text

页面上的大多数表都位于注释标记内，因此无法直接访问.

Most of the tables on the page are inside comment tags, so can't be accessed in a straightforward way.

print(soup.table.text)

返回:

1
2
3
4
OT
Final







via Sports Logos.net
About logos


New England Patriots
0
3
6
19 
6
34





via Sports Logos.net
About logos


Atlanta Falcons
0
21
7
0
0
28

即包含玩家统计信息的主表丢失了.我试图使用

i.e. the main tables containing the player stats are missing. I have tried to simply remove the comment tags using

html = html.replace('<!--',"")
html = html.replace('-->',"")

但无济于事.如何访问这些已注释掉的表?

but to no avail. How can I access these commented-out tables?

推荐答案

在这里.您可以从该页面获取任何表，只需更改索引号即可.

Here you go. You can get any table from that page only changing the index number.

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.pro-football-reference.com/boxscores/201702050atl.htm').text

soup = BeautifulSoup(page,'lxml')
table = soup.find_all('table')[1]  #This is the index of any table of that page. If you change it you can get different tables.
tab_data = [[celldata.text for celldata in rowdata.find_all(["th","td"])]
                        for rowdata in table.find_all("tr")]
for data in tab_data:
    print(' '.join(data))

由于除前两个表外的其他表都在javascript中，所以这就是为什么您需要使用硒进行崩溃和解析的原因.您现在绝对可以从该页面访问任何表.这是经过修改的.

As the other tables except for the first two are within javascript, that is why you need to use selenium to gatecrash and parse them. You will definitely be able to access any table from that page now. Here is the modified one.

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get('https://www.pro-football-reference.com/boxscores/201702050atl.htm')
soup = BeautifulSoup(driver.page_source,'lxml')
driver.quit()
table = soup.find_all('table')[7]  #This is the index of any table of that page. If you change it you can get different tables.
tab_data = [[celldata.text for celldata in rowdata.find_all(["th","td"])]
                        for rowdata in table.find_all("tr")]
for data in tab_data:
    print(' '.join(data))

这篇关于使用BeautifulSoup在评论标签内抓取表格的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！