HTML转换为可读文本 [英] HTML to readable text
问题描述
我正在编写要在Google App Engine上运行的程序.通过从HTML源中删除标记,脚本和任何其他不可读的东西(类似于nltk.clear_html),只需获取一个URL并返回文本即可.
I'm writing a program to run on Google App Engine. Which simply get an URL and return the text by removing markups, scripts and any other non-readable things from its HTML source(similar to nltk.clear_html).
HtmlTool,由 Eike .
HtmlTool by Eike.
import urllib
class HtmlTool(object):
import HTMLParser
import re
"""
Algorithms to process HTML.
"""
#Regular expressions to recognize different parts of HTML.
#Internal style sheets or JavaScript
script_sheet = re.compile(r"<(script|style).*?>.*?(</\1>)",
re.IGNORECASE | re.DOTALL)
#HTML comments - can contain ">"
comment = re.compile(r"<!--(.*?)-->", re.DOTALL)
#HTML tags: <any-text>
tag = re.compile(r"<.*?>", re.DOTALL)
#Consecutive whitespace characters
nwhites = re.compile(r"[\s]+")
#<p>, <div>, <br> tags and associated closing tags
p_div = re.compile(r"</?(p|div|br).*?>",
re.IGNORECASE | re.DOTALL)
#Consecutive whitespace, but no newlines
nspace = re.compile("[^\S\n]+", re.UNICODE)
#At least two consecutive newlines
n2ret = re.compile("\n\n+")
#A return followed by a space
retspace = re.compile("(\n )")
#For converting HTML entities to unicode
html_parser = HTMLParser.HTMLParser()
@staticmethod
def to_nice_text(html):
"""Remove all HTML tags, but produce a nicely formatted text."""
if html is None:
return u""
text = html
text = HtmlTool.script_sheet.sub(" ", text)
text = HtmlTool.comment.sub(" ", text)
text = HtmlTool.nwhites.sub(" ", text)
text = HtmlTool.p_div.sub("\n", text) #convert <p>, <div>, <br> to "\n"
text = HtmlTool.tag.sub(" ", text) #remove all tags
text = HtmlTool.html_parser.unescape(text)
#Get whitespace right
text = HtmlTool.nspace.sub(" ", text)
text = HtmlTool.retspace.sub("\n", text)
text = HtmlTool.n2ret.sub("\n\n", text)
text = text.strip()
return text
它适用于" http://google.com '
text = HtmlTool.to_nice_text(urllib.urlopen('http://google.com').read())
但是,它为' http://yahoo.com '
text = HtmlTool.to_nice_text(urllib.urlopen('http://yahoo.com').read())
错误:
Traceback (most recent call last):
File "C:\Users\BK\Desktop\Working Folder\AppEngine\crawlnsearch\-test.py", line 51, in <module>
text = HtmlTool.to_nice_text(urllib.urlopen('http://yahoo.com').read())
File "C:\Users\BK\Desktop\Working Folder\AppEngine\crawlnsearch\-test.py", line 43, in to_nice_text
text = HtmlTool.html_parser.unescape(text)
File "C:\Python27\lib\HTMLParser.py", line 472, in unescape
return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s)
File "C:\Python27\lib\re.py", line 151, in sub
return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 1531: ordinal not in range(128)
因此,任何人都可以解释这段代码有什么问题并为此发布修复程序,或者请告诉我如何使用nltk.clean_html.
So, can any one explain whats wrong with this code and post a fix for this or please tell me how to use nltk.clean_html.
推荐答案
这是因为unicode和字节串之间存在混合.
That's because there's a mix between unicode and bytestrings.
如果您在带有HtmlTool的模块中使用
If you use in the module with HtmlTool
from __future__ import unicode_literals
要确保每个" "
类块都是unicode,并且
To ensure every " "
-like blocks are unicode, and
text = HtmlTool.to_nice_text(urllib.urlopen(url).read().decode("utf-8"))
要将UTF-8字符串发送到您的方法,可以解决您的问题.
To send a UTF-8 string to your method, that solves your problem.
有关Unicode的更多信息,请阅读以下内容: http://www.joelonsoftware.com/articles/Unicode.html
Please read this for more information about Unicode: http://www.joelonsoftware.com/articles/Unicode.html
这篇关于HTML转换为可读文本的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!