nltk:如何将周围的单词词形还原? [英] nltk: How to lemmatize taking surrounding words into context?

问题描述

以下代码打印出leaf:

from nltk.stem.wordnet import WordNetLemmatizerlem = WordNetLemmatizer()打印(lem.lemmatize('叶'))

这可能准确，也可能不准确，具体取决于周围环境，例如玛丽离开房间 vs. 露珠从树叶上落下.我如何告诉 NLTK 在考虑周围上下文的情况下对单词进行词形还原?

解决方案

TL;DR

首先标记句子，然后使用词性标记作为词形还原的附加参数输入.

from nltk import pos_tag从 nltk.stem 导入 WordNetLemmatizerwnl = WordNetLemmatizer()def penn2morphy(penntag):""" 将 Penn Treebank 标签转换为 WordNet."""morphy_tag = {'NN':'n', 'JJ':'a','VB':'v', 'RB':'r'}尝试:返回 morphy_tag[penntag[:2]]除了:返回'n'def lemmatize_sent(text):# 文本输入为字符串，返回小写字符串.返回 [wnl.lemmatize(word.lower(), pos=penn2morphy(tag))对于单词，在 pos_tag(word_tokenize(text))] 中的标记lemmatize_sent('他正在走路上学')

有关如何以及为什么需要 POS 标签的详细演练，请参阅 https://www.kaggle.com/alvations/basic-nlp-with-nltk

<小时>

或者，您可以使用 pywsd tokenizer + lemmatizer，NLTK 的 WordNetLemmatizer 的包装器:

安装:

pip install -U nltkpython -m nltk.downloader 流行pip install -U pywsd

代码:

<预><代码>>>>从 pywsd.utils 导入 lemmatize_sentence预热 PyWSD(大约需要 10 秒)... 耗时 9.307677984237671 秒.>>>text = "玛丽离开房间">>>lemmatize_sentence(文本)['玛丽', '离开', 'the', '房间']>>>text = '从树叶上落下的露珠'>>>lemmatize_sentence(文本)['dew', 'drop', 'fall', 'from', 'the', 'leaf']

The following code prints out leaf:

from nltk.stem.wordnet import WordNetLemmatizer

lem = WordNetLemmatizer()
print(lem.lemmatize('leaves'))

This may or may not be accurate depending on the surrounding context, e.g. Mary leaves the room vs. Dew drops fall from the leaves. How can I tell NLTK to lemmatize words taking surrounding context into account?

解决方案

TL;DR

First tag the sentence, then use the POS tag as the additional parameter input for the lemmatization.

from nltk import pos_tag
from nltk.stem import WordNetLemmatizer

wnl = WordNetLemmatizer()

def penn2morphy(penntag):
    """ Converts Penn Treebank tags to WordNet. """
    morphy_tag = {'NN':'n', 'JJ':'a',
                  'VB':'v', 'RB':'r'}
    try:
        return morphy_tag[penntag[:2]]
    except:
        return 'n' 

def lemmatize_sent(text): 
    # Text input is string, returns lowercased strings.
    return [wnl.lemmatize(word.lower(), pos=penn2morphy(tag)) 
            for word, tag in pos_tag(word_tokenize(text))]

lemmatize_sent('He is walking to school')

For a detailed walkthrough of how and why the POS tag is necessary see https://www.kaggle.com/alvations/basic-nlp-with-nltk

---

Alternatively, you can use pywsd tokenizer + lemmatizer, a wrapper of NLTK's WordNetLemmatizer:

Install:

pip install -U nltk
python -m nltk.downloader popular
pip install -U pywsd

Code:

>>> from pywsd.utils import lemmatize_sentence
Warming up PyWSD (takes ~10 secs)... took 9.307677984237671 secs.

>>> text = "Mary leaves the room"
>>> lemmatize_sentence(text)
['mary', 'leave', 'the', 'room']

>>> text = 'Dew drops fall from the leaves'
>>> lemmatize_sentence(text)
['dew', 'drop', 'fall', 'from', 'the', 'leaf']

这篇关于nltk:如何将周围的单词词形还原?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！