如何使用 NLTK 标记器去除标点符号? [英] How to get rid of punctuation using NLTK tokenizer?
问题描述
我刚刚开始使用 NLTK,我不太明白如何从文本中获取单词列表.如果我使用 nltk.word_tokenize()
,我会得到一个单词和标点符号列表.我只需要单词.我怎样才能摆脱标点符号?此外,word_tokenize
不适用于多个句子:在最后一个单词上添加点.
I'm just starting to use NLTK and I don't quite understand how to get a list of words from text. If I use nltk.word_tokenize()
, I get a list of words and punctuation. I need only the words instead. How can I get rid of punctuation? Also word_tokenize
doesn't work with multiple sentences: dots are added to the last word.
推荐答案
看看 nltk 提供的其他标记化选项 这里.例如,您可以定义一个分词器,它挑选出字母数字字符序列作为标记并删除其他所有内容:
Take a look at the other tokenizing options that nltk provides here. For example, you can define a tokenizer that picks out sequences of alphanumeric characters as tokens and drops everything else:
from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r'w+')
tokenizer.tokenize('Eighty-seven miles to go, yet. Onward!')
输出:
['Eighty', 'seven', 'miles', 'to', 'go', 'yet', 'Onward']
这篇关于如何使用 NLTK 标记器去除标点符号?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!