当班级都一样时，如何从表的特定部分刮取? [英] How do scrape from a particular part of table when the class is all the same?
问题描述
我正在尝试从这个网站抓取数据，该网站有一个不同类别的游戏积分表.我想将总共 24 个类别放入 24 列.在示例中 "https://www.mobygames.com/developer/sheet/view/developerId,1/"有 5 个(生产、设计、工程和感谢).
I'm trying to scrape data from this website, which has a table of game credits for different categories. There are a total of 24 categories that I want to make in to 24 columns. In the example "https://www.mobygames.com/developer/sheet/view/developerId,1/" there are 5 (production, design, engineering, and thanks).
如果不同部分中的行具有不同的类，但它们都具有相同的 tr 类:devCreditsHighlight"，那将会很容易.不同的页面有不同的部分，根据页面的不同，顺序也会发生变化.最重要的是，我需要的信息从表的下一行开始，行数是随机的.
It would have been easy if rows in different sections have different class but they all have the same tr class: "devCreditsHighlight". Different page has different sections, and depending on the page the order changes too. On top of that the information that I need starts in the next row of the table and the number of rows are random.
我的问题是
有没有办法抓取包含某些关键字的表格的某个部分?例如，当您遇到文本Business"时开始抓取，然后在遇到 class="5" 时停止抓取
Is there a way to scrape a certain section of a table that includes certain keywords? for example start scraping when you encounter the text "Business" and then stop scraping when you encounter class="5"
下面是我的代码:
import bs4 as bs
import urllib.request
gameurl = "https://www.mobygames.com/developer/sheet/view/developerId,1"
req = urllib.request.Request(gameurl,headers={'User-Agent': 'Mozilla/5.0'})
sauce = urllib.request.urlopen(req).read()
soup = bs.BeautifulSoup(sauce,'lxml')
infopage = soup.find_all("div", {"class":"col-md-8 col-lg-8"})
core_list =[]
for credits in infopage:
niceHeaderTitle = credits.find_all("h1", {"class":"niceHeaderTitle"})
name = niceHeaderTitle[0].text
Titles = credits.find_all("h3", {"class":"clean"})
Titles = [title.get_text() for title in Titles]
if 'Business' in Titles:
businessinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
business = businessinfo[0].get_text(strip=True)
else:
business = 'none'
if 'Production' in Titles:
productioninfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
production = productioninfo[0].get_text(strip=True)
else:
production = 'none'
if 'Design' in Titles:
designinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
design = designinfo[0].get_text(strip=True)
else:
design = 'none'
if 'Writers' in Titles:
writersinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
writers = writersinfo[0].get_text(strip=True)
else:
writers = 'none'
if 'Writers' in Titles:
writersinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
writers = writersinfo[0].get_text(strip=True)
else:
writers = 'none'
if 'Programming/Engineering' in Titles:
programinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
program = programinfo[0].get_text(strip=True)
else:
video = 'none'
if 'Video/Cinematics' in Titles:
videoinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
video = videoinfo[0].get_text(strip=True)
else:
video = 'none'
if 'Audio' in Titles:
Audioinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
audio = Audioinfo[0].get_text(strip=True)
else:
audio = 'none'
if 'Art/Graphics' in Titles:
artinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
art = artinfo[0].get_text(strip=True)
else:
art = 'none'
if 'Support' in Titles:
supportinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
support = supportinfo[0].get_text(strip=True)
else:
support = 'none'
if 'Thanks' in Titles:
thanksinfo = credits.find_all("tr", {"class":"devCreditsHighlight"})
thanks = thanksinfo[0].get_text(strip=True)
else:
thanks = 'none'
games=[name,business,production,design,writers,video,audio,art,support,program,thanks]
core_list.append(games)
print (core_list)
这里是网页的 HTML
and here is the HTML of the webpage
</div>
</div></div>
<div class="row">
<div class="col-md-8 col-lg-8" style="overflow: hidden;">
<h1 class="niceHeaderTitle"><div class="pull-right"><a href="https://www.mobygames.com/developer/sheet/contribute/developerId,1/" class="btn btn-xs btn-mobysuccess">Contribute</a> </div>Brian Reynolds (I)</h1><ul class="nav nav-tabs" style="margin-bottom: 15px;"><li class="active"><a href="https://www.mobygames.com/developer/sheet/view/developerId,1/">Main</a></li><li><a href="https://www.mobygames.com/developer/brian-reynolds-i/credits/developerId,1/">Credits</a></li><li><a href="https://www.mobygames.com/developer/sheet/bio/developerId,1/">Biography</a></li><li><a href="https://www.mobygames.com/developer/shots/developerId,1/">Portraits</a></li></ul><h2 class="m5">Game Credits</h2>
<table class="devCreditsTable">
<tr><td colspan=5><h3 class="clean">Production</h3></td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/catan">Catan</a> (2007)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Project Lead</span>)</td></tr>
<tr><td colspan=5> </td></tr>
<tr><td colspan=5><h3 class="clean">Design</h3></td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/rise-of-nations-extended-edition">Rise of Nations: Extended Edition</a> (2017)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Design Lead</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/dominations">DomiNations</a> (2015)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Creative Director</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/age-of-empires-iii-the-asian-dynasties">Age of Empires III: The Asian Dynasties</a> (2007)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Creative Lead</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/catan">Catan</a> (2007)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Artificial Intelligence</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/rise-of-nations-rise-of-legends">Rise of Nations: Rise of Legends</a> (2006)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Design Lead</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/rise-of-nations-gold-edition">Rise of Nations: Gold Edition</a> (2004)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Design Lead</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/rise-of-nations-thrones-patriots">Rise of Nations: Thrones & Patriots</a> (2004)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Design Lead</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/rise-of-nations">Rise of Nations</a> (2003)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Game Design</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-alpha-centauri-planetary-pack">Sid Meier's Alpha Centauri: Planetary Pack</a> (2001)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Created By</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/civilization-ii-multiplayer-gold-edition">Civilization II: Multiplayer Gold Edition</a> (1999)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Game Design</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-alien-crossfire">Sid Meier's Alien Crossfire</a> (1999)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Design</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-alpha-centauri">Sid Meier's Alpha Centauri</a> (1999)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Lead Design</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-gettysburg">Sid Meier's Gettysburg!</a> (1997)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Design</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-civilization-ii">Sid Meier's Civilization II</a> (1996)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Game Design</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-colonization">Sid Meier's Colonization</a> (1994)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Game Design By</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/return-of-the-phantom">Return of the Phantom</a> (1993)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">MADS (MicroProse Adventure Development System) Designed by</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/super-quest">Super Quest</a> (1983)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Original Game Idea ('Quest 1')</span>)</td></tr>
<tr><td colspan=5> </td></tr>
<tr><td colspan=5><h3 class="clean">Programming/Engineering</h3></td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/rise-of-nations-rise-of-legends">Rise of Nations: Rise of Legends</a> (2006)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Project Lead</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-alpha-centauri-planetary-pack">Sid Meier's Alpha Centauri: Planetary Pack</a> (2001)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Additional Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/civilization-ii-multiplayer-gold-edition">Civilization II: Multiplayer Gold Edition</a> (1999)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-alien-crossfire">Sid Meier's Alien Crossfire</a> (1999)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Additional Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-alpha-centauri">Sid Meier's Alpha Centauri</a> (1999)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-gettysburg">Sid Meier's Gettysburg!</a> (1997)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-civilization-ii">Sid Meier's Civilization II</a> (1996)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/once-upon-a-forest">Once Upon a Forest</a> (1995)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">MADS Game Engine</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/dragonsphere">Dragonsphere</a> (1994)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Technical Director</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-colonization">Sid Meier's Colonization</a> (1994)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/return-of-the-phantom">Return of the Phantom</a> (1993)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Lead Programmer</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/f-15-strike-eagle-iii">F-15 Strike Eagle III</a> (1992)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Installation Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/rex-nebular-and-the-cosmic-gender-bender">Rex Nebular and the Cosmic Gender Bender</a> (1992)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Programming</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/quest-1">Quest 1</a> (1981)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">By</span>)</td></tr>
<tr><td colspan=5> </td></tr>
<tr><td colspan=5><h3 class="clean">Support</h3></td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/ville">The Ville</a> (2012)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Additional Help</span>)</td></tr>
<tr><td colspan=5> </td></tr>
<tr><td colspan=5><h3 class="clean">Thanks</h3></td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/kingdoms-of-amalur-reckoning">Kingdoms of Amalur: Reckoning</a> (2012)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Additional Thanks</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/c-evo">C-evo</a> (1999)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Game Design</span>)</td></tr>
<tr class="devCreditsHighlight">
<td><a href="https://www.mobygames.com/game/sid-meiers-civnet">Sid Meier's CivNet</a> (1995)</td><td class="devCreditsDivider"> </td><td style="white-space: nowrap;">(<span class="devCreditsTitle">Special Thanks To</span>)</td></tr>
<tr><td colspan=5> </td></tr>
</table>
现在我的结果是:
[['Contribute Brian Reynolds (I)', 'none', 'Catan(2007)(Project Lead)', 'Catan(2007)(Project Lead)', 'none', 'none', 'none', '无', 'Catan(2007)(项目负责人)', 'Catan(2007)(项目负责人)', 'Catan(2007)(项目负责人)']]
[['Contribute Brian Reynolds (I)', 'none', 'Catan(2007)(Project Lead)', 'Catan(2007)(Project Lead)', 'none', 'none', 'none', 'none', 'Catan(2007)(Project Lead)', 'Catan(2007)(Project Lead)', 'Catan(2007)(Project Lead)']]
我想要的是获取表格每个特定部分的内容.例如，第一节生产正确收集'Catan(2007)(项目负责人)'，但这只是因为它恰好在第一项中.因为所有其他人都有相同的课程，我不确定如何收集其他内容.所以它会像
What I want is to get the content in each specific sections of the table. For example The first section production is correctly collected 'Catan(2007)(Project Lead)' but its just because it happened to be in the first item. Because all the others have the same class I am not sure how to collect other content. So it would be something like
[['Contribute Brian Reynolds (I)', 'none', 'Catan(2007)(Project Lead)', 'Catan(2007)(Project Lead)', 'none', 'none', 'none', 'none', 'Rise of Nations: Extended Edition (2017) (Design Lead)DomiNations (2015)(Creative Director)Age of Empires III: The Asian Dynasties (2007) (Creative Lead)Catan (2007)(人工智能) 国家的崛起:传奇的崛起 (2006)(设计负责人)国家的崛起:黄金版(2004)(设计负责人)国家的崛起:王座 &Patriots (2004)(Design Lead)Rise of Nations (2003)(Game Design)Sid Meier's Alpha Centauri: Planetary Pack (2001) (Created By)Civilization II: Multiplayer Gold Edition (1999)(Game Design)Sid Meier's Alien Crossfire (1999)(设计)Sid Meier 的半人马座阿尔法星 (1999)(Lead Design)Sid Meier 的埃蒂斯堡！(1997) (Design)Sid Meier's Civilization II (1996)(Game Design)Sid Meier's Colonization (1994) (Game Design By)Return of the Phantom (1993) (MADS (MicroProse Adventure Development System) 设计者)Super Quest (1983))(Original Game Idea ('Quest 1'))', 'Rise of Nations: Rise of Legends (2006) (Project Lead)Sid Meier's Alpha Centauri: Planetary Pack (2001) (Additional Programming)Civilization II: Multiplayer Gold Edition (1999) (Programming)Sid Meier's Alien Crossfire (1999)(Additional Programming)Sid Meier's Alpha Centauri 1999)(Programming)Sid Meier's Gettysburg！(1997)(Programming)Sid Meier's Civilization II (1996) (Programming)Once Being a Forest (1995)(MADS Game Engine)Dragonsphere (1994)(Technical Director)Sid Meier's Colonization (1994)(Programming)幻影归来(1993)(首席程序员)F-15 Strike Eagle III (1992)(安装编程)Rex Nebular and the Cosmic Gender Bender (1992)(Programming)Quest 1 (1981) (By)', 'The Ville (2012) (Additional帮助)']]
[['Contribute Brian Reynolds (I)', 'none', 'Catan(2007)(Project Lead)', 'Catan(2007)(Project Lead)', 'none', 'none', 'none', 'none', 'Rise of Nations: Extended Edition (2017) (Design Lead)DomiNations (2015)(Creative Director)Age of Empires III: The Asian Dynasties (2007) (Creative Lead)Catan (2007)(Artificial Intelligence)Rise of Nations: Rise of Legends (2006)(Design Lead)Rise of Nations: Gold Edition (2004)(Design Lead)Rise of Nations: Thrones & Patriots (2004)(Design Lead)Rise of Nations (2003)(Game Design)Sid Meier's Alpha Centauri: Planetary Pack (2001) (Created By)Civilization II: Multiplayer Gold Edition (1999)(Game Design)Sid Meier's Alien Crossfire (1999)(Design)Sid Meier's Alpha Centauri (1999)(Lead Design)Sid Meier's ettysburg! (1997) (Design)Sid Meier's Civilization II (1996)(Game Design)Sid Meier's Colonization (1994) (Game Design By)Return of the Phantom (1993) (MADS (MicroProse Adventure Development System) Designed by)Super Quest (1983)(Original Game Idea ('Quest 1'))', 'Rise of Nations: Rise of Legends (2006) (Project Lead)Sid Meier's Alpha Centauri: Planetary Pack (2001) (Additional Programming)Civilization II: Multiplayer Gold Edition (1999) (Programming)Sid Meier's Alien Crossfire (1999)(Additional Programming)Sid Meier's Alpha Centauri 1999)(Programming)Sid Meier's Gettysburg! (1997)(Programming)Sid Meier's Civilization II (1996) (Programming)Once Upon a Forest (1995)(MADS Game Engine)Dragonsphere (1994)(Technical Director)Sid Meier's Colonization (1994)(Programming)Return of the Phantom (1993)(Lead Programmer)F-15 Strike Eagle III (1992) (Installation Programming)Rex Nebular and the Cosmic Gender Bender (1992)(Programming)Quest 1 (1981) (By)', 'The Ville (2012) (Additional Help)']]
推荐答案
我知道这会有多困难，格式很难处理，因为标题不是该部分的父级，但我找到了一种方法大约.
I can see how this would be difficult, the format is quite hard to deal with because the title isn't the parent of the section but I found a way around.
现在不只是第一行拉取所有行
now pulls all rows not just first
import bs4 as bs
import urllib.request
import numpy as np
gameurl = "https://www.mobygames.com/developer/sheet/view/developerId,1"
req = urllib.request.Request(gameurl,headers={'User-Agent': 'Mozilla/5.0'})
sauce = urllib.request.urlopen(req).read()
soup = bs.BeautifulSoup(sauce,'lxml')
infopage = soup.find_all("div", {"class":"col-md-8 col-lg-8"})
core_list =[]
for credits in infopage:
niceHeaderTitle = credits.find_all("h1", {"class":"niceHeaderTitle"})
name = niceHeaderTitle[0].text
Titles = credits.find_all("h3", {"class":"clean"})
Titles = [title.get_text() for title in Titles]
tr = credits.find_all("tr")
for i in range(len(tr)):
row = tr[i].get_text()
if row in Titles:
title = row
elif len(row) > 1:
games=[name,title,row]
core_list.append(games)
core_list = np.matrix(core_list)
要将其放入列中，因为每个人都有一组不同的类别，您可能应该像这样提取它，添加所有其他人，然后使用一些数据框操作(枢轴)将其放入 24 列你要找的
To get it into columns, because each person will have a different set of categories you should probably extract it like this, get all the other people added and then use some data frame manipulation (pivot) to make it into the 24 columns you're looking for
这篇关于当班级都一样时，如何从表的特定部分刮取?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
