如何使用NLTK标记器消除标点符号? [英] How to get rid of punctuation using NLTK tokenizer?

问题描述

我刚刚开始使用NLTK，但我不太了解如何从文本中获取单词列表.如果使用nltk.word_tokenize()，则会得到单词和标点的列表.我只需要这些词.我如何摆脱标点符号?同样，word_tokenize不适用于多个句子:在最后一个单词上加点.

I'm just starting to use NLTK and I don't quite understand how to get a list of words from text. If I use nltk.word_tokenize(), I get a list of words and punctuation. I need only the words instead. How can I get rid of punctuation? Also word_tokenize doesn't work with multiple sentences: dots are added to the last word.

推荐答案

看看nltk提供的其他标记化选项

Take a look at the other tokenizing options that nltk provides here. For example, you can define a tokenizer that picks out sequences of alphanumeric characters as tokens and drops everything else:

from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r'\w+')
tokenizer.tokenize('Eighty-seven miles to go, yet.  Onward!')

输出:

['Eighty', 'seven', 'miles', 'to', 'go', 'yet', 'Onward']

这篇关于如何使用NLTK标记器消除标点符号?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！