如何使用BeautifulSoup从网页上获取整个正文? [英] How can I grab the entire body text from a web page using BeautifulSoup?

问题描述

我想从一个自然语言处理项目的医学文档的网页上获取一些文本，并且在使用BeautifulSoup提取必要的信息时遇到问题.我正在查看的网站可以在以下地址找到:

I would like to grab some text from a webpage of a medical document for a Natural Language Processing project and am having issues extracting the necessary information using BeautifulSoup. The website I am viewing can be found at the address: https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D

我想做的是从此页面上抓取整个文本主体，并使用光标进行操作，只需应用复制/粘贴即可为我提供我感兴趣的适当文本:

What I would like to do is grab the entire text body from this page and doing so with my cursor and simply applying a copy/paste would give me the appropriate text I am interested in:

Sample Type / Medical Specialty: Gastroenterology
Sample Name: Abdominal Abscess I&D
Description: Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.
(Medical Transcription Sample Report)
PREOPERATIVE DIAGNOSIS: Abdominal wall abscess.

... (body text) ...

The finished wound size was 9.0 x 5.3 x 5.2 cm in size. Patient tolerated the procedure well. Dressing was applied, and he was taken to recovery room in stable condition. 

但是，我想使用BeautifulSoup来实现此目的，因为我想执行一个循环以从同一网站获取多个医疗文档.

However, I would like to implement this using BeautifulSoup because I would like to perform a loop to grab multiple medical documents from the same website.

import requests  
r = requests.get('https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D')

from bs4 import BeautifulSoup  
soup = BeautifulSoup(r.text, 'html.parser')  
results = soup.find_all('div', attrs={'id':'sampletext'})

# Here I am able to specify the <h1> tag to get 'Sample Type / Medical Specialty' as well as 'Sample Name' text fields

record.find('h1').text.replace('\n', ' ')

但是，我无法为其余文本(即说明，术前诊断，术后诊断，手术等)复制此文本，因为没有唯一的标签来标识这些文本字段

However, I cannot replicate this for the remaining text (i.e. Description, PREOPERATIVE DIAGNOSIS, POSTOPERATIVE DIAGNOSIS, Procedure, etc.) as there are no unique tags to identify these text fields

如果任何人都熟悉使用BeautifulSoup进行网络抓取的概念，我将不胜感激！同样，我的目标是从网页中获取全文，最终将其添加到Pandas Dataframe中.谢谢！

If anyone is familiar with web-scraping concepts using BeautifulSoup I would appreciate any feedback! Again my goal is to obtain the full text from the webpage which I would ultimately like to add to a Pandas Dataframe. Thanks!

推荐答案

好，我花了一些时间，但是没有一种简单的方法来提取可用的文本，除非您手动遍历所有元素:

Ok, it took me a while, but there isn't an easy way of extracting usable text unless you manually iterate over all elements:

import requests
import re
from bs4 import BeautifulSoup, Tag, NavigableString, Comment

url = 'https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D'
res = requests.get(url)
res.raise_for_status()
html = res.text
soup = BeautifulSoup(html, 'html.parser')

到目前为止没有什么特别的.

so far nothing special.

title_el = soup.find('h1')
page_title = title_el.text.strip()
first_hr = title_el.find_next_sibling('hr')

description_title = title_el.find_next_sibling('b', text=re.compile('description', flags=re.I))
description_text_parts = []
for s in description_title.next_siblings:
    if s is first_hr:
        break
    if isinstance(s, Tag):
        description_text_parts.append(s.text.strip())
    elif isinstance(s, NavigableString):
        description_text_parts.append(str(s).strip())
description_text = '\n'.join(p for p in description_text_parts if p.strip())

在这里，我们从< h1>

'Sample Type / Medical Specialty:  Gastroenterology\nSample Name: Abdominal Abscess I&D'

在看到文本 Description:之后，通过遍历元素来

和 description .

and description by walking the elements after we see the text Description:.

'Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.\n(Medical Transcription Sample Report)'

现在，所有标题均置于水平线以下:

Now, all titles are placed under the horizontal rule:

# titles are all bold and uppercase
titles = [b for b in first_hr.find_next_siblings('b') if b.text.strip().isupper()]

我们在标题之间找到文本，并将其分配给我们之前看到的标题

We find the text between the titles and assign it to the title we see earlier

docs = []
for t in titles:
    text_parts = []
    for s in t.next_siblings:
        # go until next title
        if s in titles:
            break
        if isinstance(s, Comment):
            continue
        if isinstance(s, Tag):
            if s.name == 'div':
                break
            text_parts.append(s.text.strip())
        elif isinstance(s, NavigableString):
            text_parts.append(str(s).strip())
    text = '\n'.join(p for p in text_parts if p.strip())
    docs.append({
        'title': t.text.strip(),
        'text': text
    })

打印文档给出:

[
{'title': 'PREOPERATIVE DIAGNOSIS:', 'text': 'Abdominal wall abscess.'}, 
{'title': 'POSTOPERATIVE DIAGNOSIS:', 'text': 'Abdominal wall abscess.'}, 
{'title': 'PROCEDURE:', 'text': 'Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.'}, 
{'title': 'ANESTHESIA:', 'text': 'LMA.'}, 
{'title': 'INDICATIONS:', 'text': 'Patient is a pleasant 60-year-old gentleman, who initially had a sigmoid colectomy for diverticular abscess, subsequently had a dehiscence with evisceration.  Came in approximately 36 hours ago with pain across his lower abdomen.  CT scan demonstrated presence of an abscess beneath the incision.  I recommended to the patient he undergo the above-named procedure.  Procedure, purpose, risks, expected benefits, potential complications, alternatives forms of therapy were discussed with him, and he was agreeable to surgery.'}, 
{'title': 'FINDINGS:', 'text': 'The patient was found to have an abscess that went down to the level of the fascia.  The anterior layer of the fascia was fibrinous and some portions necrotic.  This was excisionally debrided using the Bovie cautery, and there were multiple pieces of suture within the wound and these were removed as well.'},
{'title': 'TECHNIQUE:', 'text': 'Patient was identified, then taken into the operating room, where after induction of appropriate anesthesia, his abdomen was prepped with Betadine solution and draped in a sterile fashion.  The wound opening where it was draining was explored using a curette.  The extent of the wound marked with a marking pen and using the Bovie cautery, the abscess was opened and drained.  I then noted that there was a significant amount of undermining.  These margins were marked with a marking pen, excised with Bovie cautery; the curette was used to remove the necrotic fascia.  The wound was irrigated; cultures sent prior to irrigation and after achievement of excellent hemostasis, the wound was packed with antibiotic-soaked gauze.  A dressing was applied.  The finished wound size was 9.0 x 5.3 x 5.2 cm in size.  Patient tolerated the procedure well.  Dressing was applied, and he was taken to recovery room in stable condition.'}
]

这篇关于如何使用BeautifulSoup从网页上获取整个正文?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！