Python NLTK pos_tag 未返回正确的词性标记 [英] Python NLTK pos_tag not returning the correct part-of-speech tag

问题描述

有这个:

text = word_tokenize("敏捷的棕色狐狸跳过懒狗")

并运行:

nltk.pos_tag(text)

我明白了:

[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('跳转', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

这是不正确的.句子中quick brown lazy的标签应该是:

('quick', 'JJ'), ('brown', 'JJ') , ('懒惰', 'JJ')

通过他们的在线工具测试得到相同的结果；quick、brown 和 fox 应该是形容词而不是名词.

解决方案

简而言之:

<块引用>

NLTK 并不完美.事实上，没有任何模型是完美的.

注意:

从 NLTK 3.1 版开始，默认的 pos_tag 函数不再是 旧的 MaxEnt 英语泡菜.

现在是 @Honnibal 的实现，见nltk.tag.pos_tag

<预><代码>>>>进口检验>>>打印inspect.getsource(pos_tag)def pos_tag(tokens, tagset=None):标记器 = PerceptronTagger()return _pos_tag(tokens, tagset, tagger)

仍然更好但并不完美:

<预><代码>>>>从 nltk 导入 pos_tag>>>pos_tag("敏捷的棕色狐狸跳过懒狗".split())[('The', 'DT'), ('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('跳跃', 'VBZ'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

在某些时候，如果有人想要 TL;DR 解决方案，请参阅 https://github.com/alvations/nltk_cli

<小时>

长期:

尝试使用其他标记器(参见 https://github.com/nltk/nltk/tree/develop/nltk/tag) ，例如:

• HunPos
• 斯坦福销售点
• 塞纳

使用来自 NLTK 的默认 MaxEnt POS 标记器，即 nltk.pos_tag:

<预><代码>>>>从 nltk 导入 word_tokenize, pos_tag>>>text = "敏捷的棕色狐狸跳过懒狗">>>pos_tag(word_tokenize(text))[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

使用斯坦福词性标注器:

$ cd ~$ wget http://nlp.stanford.edu/software/stanford-postagger-2015-04-20.zip$ 解压 stanford-postagger-2015-04-20.zip$ mv stanford-postagger-2015-04-20 stanford-postagger$蟒蛇>>>从 os.path 导入 expanduser>>>home = expanduser("~")>>>从 nltk.tag.stanford 导入 POSTagger>>>_path_to_model = home + '/stanford-postagger/models/english-bidirectional-distim.tagger'>>>_path_to_jar = 家 + '/stanford-postagger/stanford-postagger.jar'>>>st = POSTagger(path_to_model=_path_to_model, path_to_jar=_path_to_jar)>>>text = "敏捷的棕色狐狸跳过懒狗">>>st.tag(text.split())[(u'The', u'DT'), (u'quick', u'JJ'), (u'brown', u'JJ'), (u'fox', u'NN'), (u'jumps', u'VBZ'), (u'over', u'IN'), (u'the', u'DT'), (u'lazy', u'JJ'), (u'狗', u'NN')]

使用 HunPOS(注意:默认编码是 ISO-8859-1 而不是 UTF8):

$ cd ~$ wget https://hunpos.googlecode.com/files/hunpos-1.0-linux.tgz$ tar zxvf hunpos-1.0-linux.tgz$ wget https://hunpos.googlecode.com/files/en_wsj.model.gz$ gzip -d en_wsj.model.gz$ mv en_wsj.model hunpos-1.0-linux/$蟒蛇>>>从 os.path 导入 expanduser>>>home = expanduser("~")>>>从 nltk.tag.hunpos 导入 HunposTagger>>>_path_to_bin = 家 + '/hunpos-1.0-linux/hunpos-tag'>>>_path_to_model = home + '/hunpos-1.0-linux/en_wsj.model'>>>ht = HunposTagger(path_to_model=_path_to_model, path_to_bin=_path_to_bin)>>>text = "敏捷的棕色狐狸跳过懒狗">>>ht.tag(text.split())[('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

使用 Senna(确保您拥有最新版本的 NLTK，对 API 进行了一些更改):

$ cd ~$ wget http://ronan.collobert.com/senna/senna-v3.0.tgz$ tar zxvf 塞纳-v3.0.tgz$蟒蛇>>>从 os.path 导入 expanduser>>>home = expanduser("~")>>>从 nltk.tag.senna 导入 SennaTagger>>>st = SennaTagger(home+'/senna')>>>text = "敏捷的棕色狐狸跳过懒狗">>>st.tag(text.split())[('The', u'DT'), ('quick', u'JJ'), ('brown', u'JJ'), ('fox', u'NN'), ('jumps',u'VBZ'), ('over', u'IN'), ('the', u'DT'), ('lazy', u'JJ'), ('dog', u'NN')]

<小时>

或者尝试构建更好的 POS 标记器:

• Ngram 标记器:http://streamhacker.com/2008/11/03/part-of-speech-tagging-with-nltk-part-1/
• 词缀/正则表达式标记器:http://streamhacker.com/2008/11/10/part-of-speech-tagging-with-nltk-part-2/
• 构建您自己的 Brill(阅读代码，这是一个非常有趣的标记器，http://www.nltk.org/_modules/nltk/tag/brill.html)，见http://streamhacker.com/2008/12/03/part-of-speech-tagging-with-nltk-part-3/
• 感知器标记器:https://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/
• LDA 标记器:http://scm.io/blog/hack/2015/02/lda-意向/

<小时>

抱怨 pos_tag 在 stackoverflow 上的准确性包括:

• 词性标注 - NLTK 认为名词是形容词
• python NLTK POS 标记器未按预期运行
• 如何使用 NLTK pos 标签获得更好的结果
• NLTK 中的 pos_tag 不能正确标记句子

关于 NLTK HunPos 的问题包括:

• 如何在 nltk 中使用 hunpos 标记文本文件?
• 有谁知道如何在 nltk 上配置 hunpos 包装类?

NLTK 和斯坦福词性标注器的问题包括:

• 无法将 stanford pos tagger 导入 nltk
• Java 命令在 NLTK 斯坦福 POS 标记器中失败
• 在 NLTK Python 中使用 Stanford POS Tagger 时出错
• 如何使用 Stanford NLP Tagger 和NLTK
• Nltk stanford pos tagger 错误:Java 命令失败
• 在 NLTK 中实例化和使用 StanfordTagger
• 正在运行NLTK 中的斯坦福 POS 标记器导致不是有效的 Win32 应用程序"在 Windows 上

Having this:

text = word_tokenize("The quick brown fox jumps over the lazy dog")

And running:

nltk.pos_tag(text)

I get:

[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

This is incorrect. The tags for quick brown lazy in the sentence should be:

('quick', 'JJ'), ('brown', 'JJ') , ('lazy', 'JJ')

Testing this through their online tool gives the same result; quick, brown and fox should be adjectives not nouns.

解决方案

In short:

NLTK is not perfect. In fact, no model is perfect.

Note:

As of NLTK version 3.1, default pos_tag function is no longer the old MaxEnt English pickle.

It is now the perceptron tagger from @Honnibal's implementation, see nltk.tag.pos_tag

>>> import inspect
>>> print inspect.getsource(pos_tag)
def pos_tag(tokens, tagset=None):
    tagger = PerceptronTagger()
    return _pos_tag(tokens, tagset, tagger) 

Still it's better but not perfect:

>>> from nltk import pos_tag
>>> pos_tag("The quick brown fox jumps over the lazy dog".split())
[('The', 'DT'), ('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'VBZ'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

At some point, if someone wants TL;DR solutions, see https://github.com/alvations/nltk_cli

---

In long:

Try using other tagger (see https://github.com/nltk/nltk/tree/develop/nltk/tag) , e.g.:

• HunPos
• Stanford POS
• Senna

Using default MaxEnt POS tagger from NLTK, i.e. nltk.pos_tag:

>>> from nltk import word_tokenize, pos_tag
>>> text = "The quick brown fox jumps over the lazy dog"
>>> pos_tag(word_tokenize(text))
[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

Using Stanford POS tagger:

$ cd ~
$ wget http://nlp.stanford.edu/software/stanford-postagger-2015-04-20.zip
$ unzip stanford-postagger-2015-04-20.zip
$ mv stanford-postagger-2015-04-20 stanford-postagger
$ python
>>> from os.path import expanduser
>>> home = expanduser("~")
>>> from nltk.tag.stanford import POSTagger
>>> _path_to_model = home + '/stanford-postagger/models/english-bidirectional-distsim.tagger'
>>> _path_to_jar = home + '/stanford-postagger/stanford-postagger.jar'
>>> st = POSTagger(path_to_model=_path_to_model, path_to_jar=_path_to_jar)
>>> text = "The quick brown fox jumps over the lazy dog"
>>> st.tag(text.split())
[(u'The', u'DT'), (u'quick', u'JJ'), (u'brown', u'JJ'), (u'fox', u'NN'), (u'jumps', u'VBZ'), (u'over', u'IN'), (u'the', u'DT'), (u'lazy', u'JJ'), (u'dog', u'NN')]

Using HunPOS (NOTE: the default encoding is ISO-8859-1 not UTF8):

$ cd ~
$ wget https://hunpos.googlecode.com/files/hunpos-1.0-linux.tgz
$ tar zxvf hunpos-1.0-linux.tgz
$ wget https://hunpos.googlecode.com/files/en_wsj.model.gz
$ gzip -d en_wsj.model.gz 
$ mv en_wsj.model hunpos-1.0-linux/
$ python
>>> from os.path import expanduser
>>> home = expanduser("~")
>>> from nltk.tag.hunpos import HunposTagger
>>> _path_to_bin = home + '/hunpos-1.0-linux/hunpos-tag'
>>> _path_to_model = home + '/hunpos-1.0-linux/en_wsj.model'
>>> ht = HunposTagger(path_to_model=_path_to_model, path_to_bin=_path_to_bin)
>>> text = "The quick brown fox jumps over the lazy dog"
>>> ht.tag(text.split())
[('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

Using Senna (Make sure you've the latest version of NLTK, there were some changes made to the API):

$ cd ~
$ wget http://ronan.collobert.com/senna/senna-v3.0.tgz
$ tar zxvf senna-v3.0.tgz
$ python
>>> from os.path import expanduser
>>> home = expanduser("~")
>>> from nltk.tag.senna import SennaTagger
>>> st = SennaTagger(home+'/senna')
>>> text = "The quick brown fox jumps over the lazy dog"
>>> st.tag(text.split())
[('The', u'DT'), ('quick', u'JJ'), ('brown', u'JJ'), ('fox', u'NN'), ('jumps', u'VBZ'), ('over', u'IN'), ('the', u'DT'), ('lazy', u'JJ'), ('dog', u'NN')]

---

Or try building a better POS tagger:

• Ngram Tagger: http://streamhacker.com/2008/11/03/part-of-speech-tagging-with-nltk-part-1/
• Affix/Regex Tagger: http://streamhacker.com/2008/11/10/part-of-speech-tagging-with-nltk-part-2/
• Build Your Own Brill (Read the code it's a pretty fun tagger, http://www.nltk.org/_modules/nltk/tag/brill.html), see http://streamhacker.com/2008/12/03/part-of-speech-tagging-with-nltk-part-3/
• Perceptron Tagger: https://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/
• LDA Tagger: http://scm.io/blog/hack/2015/02/lda-intentions/

---

Complains about pos_tag accuracy on stackoverflow include:

• POS tagging - NLTK thinks noun is adjective
• python NLTK POS tagger not behaving as expected
• How to obtain better results using NLTK pos tag
• pos_tag in NLTK does not tag sentences correctly

Issues about NLTK HunPos include:

• How do I tag textfiles with hunpos in nltk?
• Does anyone know how to configure the hunpos wrapper class on nltk?

Issues with NLTK and Stanford POS tagger include:

• trouble importing stanford pos tagger into nltk
• Java Command Fails in NLTK Stanford POS Tagger
• Error using Stanford POS Tagger in NLTK Python
• How to improve speed with Stanford NLP Tagger and NLTK
• Nltk stanford pos tagger error : Java command failed
• Instantiating and using StanfordTagger within NLTK
• Running Stanford POS tagger in NLTK leads to "not a valid Win32 application" on Windows

这篇关于Python NLTK pos_tag 未返回正确的词性标记的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！