标记HTML文档 [英] Tokenizing an HTML document
本文介绍了标记HTML文档的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个HTML文档,我想使用spaCy对它进行标记化,同时将HTML标签保留为单个标记.这是我的代码:
I have an HTML document and I'd like to tokenize it using spaCy while keeping HTML tags as a single token. Here's my code:
import spacy
from spacy.symbols import ORTH
nlp = spacy.load('en', vectors=False, parser=False, entity=False)
nlp.tokenizer.add_special_case(u'<i>', [{ORTH: u'<i>'}])
nlp.tokenizer.add_special_case(u'</i>', [{ORTH: u'</i>'}])
doc = nlp('Hello, <i>world</i> !')
print([e.text for e in doc])
输出为:
['Hello', ',', '<', 'i', '>', 'world</i', '>', '!']
如果我在标签周围放置空格,就像这样:
If I put spaces around the tags, like this:
doc = nlp('Hello, <i> world </i> !')
输出是我想要的:
['Hello', ',', '<i>', 'world', '</i>', '!']
但是我想避免对HTML进行复杂的预处理.
but I'd like avoiding complicated pre-processing to the HTML.
任何想法我该如何处理?
Any idea how can I approach this?
推荐答案
您需要创建自定义令牌生成器.
You need to create a custom Tokenizer.
您的自定义标记生成器将与spaCy的标记生成器完全相同,但它将具有<"和'>'符号从前缀和后缀中删除,还会添加一个新的前缀和一个新的后缀规则.
Your custom Tokenizer will be exactly as spaCy's tokenizer but it will have '<' and '>' symbols removed from prefixes and suffixes and also it will add one new prefix and one new suffix rule.
代码:
import spacy
from spacy.tokens import Token
Token.set_extension('tag', default=False)
def create_custom_tokenizer(nlp):
from spacy import util
from spacy.tokenizer import Tokenizer
from spacy.lang.tokenizer_exceptions import TOKEN_MATCH
prefixes = nlp.Defaults.prefixes + ('^<i>',)
suffixes = nlp.Defaults.suffixes + ('</i>$',)
# remove the tag symbols from prefixes and suffixes
prefixes = list(prefixes)
prefixes.remove('<')
prefixes = tuple(prefixes)
suffixes = list(suffixes)
suffixes.remove('>')
suffixes = tuple(suffixes)
infixes = nlp.Defaults.infixes
rules = nlp.Defaults.tokenizer_exceptions
token_match = TOKEN_MATCH
prefix_search = (util.compile_prefix_regex(prefixes).search)
suffix_search = (util.compile_suffix_regex(suffixes).search)
infix_finditer = (util.compile_infix_regex(infixes).finditer)
return Tokenizer(nlp.vocab, rules=rules,
prefix_search=prefix_search,
suffix_search=suffix_search,
infix_finditer=infix_finditer,
token_match=token_match)
nlp = spacy.load('en_core_web_sm')
tokenizer = create_custom_tokenizer(nlp)
nlp.tokenizer = tokenizer
doc = nlp('Hello, <i>world</i> !')
print([e.text for e in doc])
这篇关于标记HTML文档的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文