NLTK WordNetLemmatizer:未实现预期的最小化 [英] NLTK WordNetLemmatizer: Not Lemmatizing as Expected
问题描述
我正在尝试使用NLTK的WordNetLemmatizer对句子中的所有单词进行词法化.我有一堆句子,但只是使用第一句话来确保我正确地做到了.这就是我所拥有的:
I'm trying to lemmatize all of the words in a sentence with NLTK's WordNetLemmatizer. I have a bunch of sentences but am just using the first sentence to ensure I'm doing this correctly. Here's what I have:
train_sentences[0]
"Explanation Why edits made username Hardcore Metallica Fan reverted? They vandalisms, closure GAs I voted New York Dolls FAC. And please remove template talk page since I'm retired now.89.205.38.27"
因此,现在我尝试如下对每个词进行词素化:
So now I try to lemmatize each word as follows:
lemmatizer = WordNetLemmatizer()
new_sent = [lemmatizer.lemmatize(word) for word in train_sentences[0].split()]
print(new_sent)
然后我回来:
['Explanation', 'Why', 'edits', 'made', 'username', 'Hardcore', 'Metallica', 'Fan', 'reverted?', 'They', 'vandalisms,', 'closure', 'GAs', 'I', 'voted', 'New', 'York', 'Dolls', 'FAC.', 'And', 'please', 'remove', 'template', 'talk', 'page', 'since', "I'm", 'retired', 'now.89.205.38.27']
几个问题:
1)为什么编辑"不转换为编辑"?诚然,如果我做lemmatizer.lemmatize("edits")
,我又回来了edits
,但感到惊讶.
1) Why does "edits" not get transformed into "edit"? Admittedly, if I do lemmatizer.lemmatize("edits")
I get back edits
but was surprised.
2)为什么故意破坏"没有转变为故意破坏"?这是非常令人惊讶的,因为如果我执行lemmatizer.lemmatize("vandalisms")
,我会返回vandalism
...
2) Why is "vandalisms" not transformed into "vandalism"? This one is very surprising, since if I do lemmatizer.lemmatize("vandalisms")
, I get back vandalism
...
任何澄清/指导都很棒!
Any clarification / guidance would be awesome!
推荐答案
TL; DR
首先标记句子,然后使用POS标记作为用于词法化的附加参数输入.
TL;DR
First tag the sentence, then use the POS tag as the additional parameter input for the lemmatization.
from nltk import pos_tag
from nltk.stem import WordNetLemmatizer
wnl = WordNetLemmatizer()
def penn2morphy(penntag):
""" Converts Penn Treebank tags to WordNet. """
morphy_tag = {'NN':'n', 'JJ':'a',
'VB':'v', 'RB':'r'}
try:
return morphy_tag[penntag[:2]]
except:
return 'n'
def lemmatize_sent(text):
# Text input is string, returns lowercased strings.
return [wnl.lemmatize(word.lower(), pos=penn2morphy(tag))
for word, tag in pos_tag(word_tokenize(text))]
lemmatize_sent('He is walking to school')
有关如何以及为何需要POS标签的详细演练,请参见 https ://www.kaggle.com/alvations/basic-nlp-with-nltk
For a detailed walkthrough of how and why the POS tag is necessary see https://www.kaggle.com/alvations/basic-nlp-with-nltk
或者,您可以使用pywsd
标记器+ lemmatizer,这是NLTK WordNetLemmatizer
的包装:
Alternatively, you can use pywsd
tokenizer + lemmatizer, a wrapper of NLTK's WordNetLemmatizer
:
安装:
pip install -U nltk
python -m nltk.downloader popular
pip install -U pywsd
代码:
>>> from pywsd.utils import lemmatize_sentence
Warming up PyWSD (takes ~10 secs)... took 9.307677984237671 secs.
>>> text = "Mary leaves the room"
>>> lemmatize_sentence(text)
['mary', 'leave', 'the', 'room']
>>> text = 'Dew drops fall from the leaves'
>>> lemmatize_sentence(text)
['dew', 'drop', 'fall', 'from', 'the', 'leaf']
(Note to moderators: I can't mark this question as duplicate of nltk: How to lemmatize taking surrounding words into context? because the answer wasn't accepted there but it is a duplicate).
这篇关于NLTK WordNetLemmatizer:未实现预期的最小化的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!