NLTK关系提取不返回任何内容 [英] NLTK relation extraction returns nothing

问题描述

我最近正在使用nltk从文本中提取关系.所以我编写了一个示例文本:汤姆是微软的联合创始人."并使用以下程序进行测试并返回任何内容.我不知道为什么.

I am recently working on using nltk to extract relation from text. so i build a sample text:" Tom is the cofounder of Microsoft." and using following program to test and return nothing. I cannot figure out why.

我正在使用NLTK版本:3.2.1，python版本:3.5.2.

I'm using NLTK version: 3.2.1, python version: 3.5.2.

这是我的代码:

import re
import nltk
from nltk.sem.relextract import extract_rels, rtuple
from nltk.tokenize import sent_tokenize, word_tokenize


def test():
    with open('sample.txt', 'r') as f:
        sample = f.read()   # "Tom is the cofounder of Microsoft"

    sentences = sent_tokenize(sample)
    tokenized_sentences = [word_tokenize(sentence) for sentence in sentences]
    tagged_sentences = [nltk.tag.pos_tag(sentence) for sentence in tokenized_sentences]

    OF = re.compile(r'.*\bof\b.*')

    for i, sent in enumerate(tagged_sentences):
        sent = nltk.chunk.ne_chunk(sent) # ne_chunk method expects one tagged sentence
        rels = extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10) 
        for rel in rels:
            print('{0:<5}{1}'.format(i, rtuple(rel)))

if __name__ == '__main__':
    test()

---

1.经过一些调试后，如果发现我将输入更改为

盖茨于1955年10月28日出生于华盛顿州西雅图."

"Gates was born in Seattle, Washington on October 28, 1955. "

nltk.chunk.ne_chunk()输出为:

(S (PERSON Gates/NNS) was/VBD 出生/VBN in/IN (GPE西雅图/NNP) ，/， (GPE华盛顿/NNP) 开/IN 十月/NNP 28/CD ，/， 1955年/CD ./.)

(S (PERSON Gates/NNS) was/VBD born/VBN in/IN (GPE Seattle/NNP) ,/, (GPE Washington/NNP) on/IN October/NNP 28/CD ,/, 1955/CD ./.)

test()返回:

[PER:'Gates/NNS']'是/VBD诞生/VBN in/IN'[GPE:'Seattle/NNP']

[PER: 'Gates/NNS'] 'was/VBD born/VBN in/IN' [GPE: 'Seattle/NNP']

2.在我将输入更改为:

盖茨于1955年10月28日出生于西雅图."

"Gates was born in Seattle on October 28, 1955. "

test()不会恢复任何作用.

The test() retuns nothing.

输出是由以下功能引起的: semi_rel2reldict(pairs，window = 5，trace = False)，仅当len(pairs)> 2时才返回结果，这就是为什么一个少于三个NE的句子将返回N​​one的原因.

output is caused by function: semi_rel2reldict(pairs, window=5, trace=False), which returns result only when len(pairs) > 2, and that's why when one sentence with less than three NEs will return None.

这是错误还是我以错误的方式使用了NLTK?

推荐答案

首先，要使用ne_chunk对网元进行分块，这个习惯用法看起来像这样

Firstly, to chunk NEs with ne_chunk, the idiom would look something like this

>>> from nltk import ne_chunk, pos_tag, word_tokenize
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')])])

(另请参见 https://stackoverflow.com/a/31838373/610569 )

接下来，让我们看一下 extract_rels函数.

Next let's look at the extract_rels function.

def extract_rels(subjclass, objclass, doc, corpus='ace', pattern=None, window=10):
    """
    Filter the output of ``semi_rel2reldict`` according to specified NE classes and a filler pattern.
    The parameters ``subjclass`` and ``objclass`` can be used to restrict the
    Named Entities to particular types (any of 'LOCATION', 'ORGANIZATION',
    'PERSON', 'DURATION', 'DATE', 'CARDINAL', 'PERCENT', 'MONEY', 'MEASURE').
    """

调用此功能时:

extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10)

它顺序执行4个过程.

即 https://github.com/nltk/nltk/blob /develop/nltk/sem/relextract.py#L202 :

if subjclass and subjclass not in NE_CLASSES[corpus]:
    if _expand(subjclass) in NE_CLASSES[corpus]:
        subjclass = _expand(subjclass)
    else:
        raise ValueError("your value for the subject type has not been recognized: %s" % subjclass)
if objclass and objclass not in NE_CLASSES[corpus]:
    if _expand(objclass) in NE_CLASSES[corpus]:
        objclass = _expand(objclass)
    else:
        raise ValueError("your value for the object type has not been recognized: %s" % objclass)

2.它从NE标签输入中提取对":

if corpus == 'ace' or corpus == 'conll2002':
    pairs = tree2semi_rel(doc)
elif corpus == 'ieer':
    pairs = tree2semi_rel(doc.text) + tree2semi_rel(doc.headline)
else:
    raise ValueError("corpus type not recognized")

现在让我们看一下输入句子Tom is the cofounder of Microsoft，tree2semi_rel()返回什么:

Now let's see given your input sentence Tom is the cofounder of Microsoft, what does tree2semi_rel() returns:

>>> from nltk.sem.relextract import tree2semi_rel, semi_rel2reldict
>>> from nltk import word_tokenize, pos_tag, ne_chunk
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]

因此它返回一个包含2个列表的列表，第一个内部列表由一个空白列表组成，而Tree包含"PERSON"标记.

So it returns a list of 2 lists, the first inner list consist of a blank list and the Tree that contains the "PERSON" tag.

[[], Tree('PERSON', [('Tom', 'NNP')])] 

第二个列表由短语is the cofounder of和包含"ORGANIZATION"的Tree组成.

The second list consist of the phrase is the cofounder of and the Tree that contains "ORGANIZATION".

我们继续前进.

reldicts = semi_rel2reldict(pairs)

如果我们用您的例句查看semi_rel2reldict函数返回的内容，我们会看到这是空列表返回的地方:

If we look what the semi_rel2reldict function returns with your example sentence, we see that this is where the empty list gets returns:

>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[]

因此，让我们看一下semi_rel2reldict的代码 https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L144 :

So let's look into the code of semi_rel2reldict https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L144:

def semi_rel2reldict(pairs, window=5, trace=False):
    """
    Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
    stores information about the subject and object NEs plus the filler between them.
    Additionally, a left and right context of length =< window are captured (within
    a given input sentence).
    :param pairs: a pair of list(str) and ``Tree``, as generated by
    :param window: a threshold for the number of items to include in the left and right context
    :type window: int
    :return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
    :rtype: list(defaultdict)
    """
    result = []
    while len(pairs) > 2:
        reldict = defaultdict(str)
        reldict['lcon'] = _join(pairs[0][0][-window:])
        reldict['subjclass'] = pairs[0][1].label()
        reldict['subjtext'] = _join(pairs[0][1].leaves())
        reldict['subjsym'] = list2sym(pairs[0][1].leaves())
        reldict['filler'] = _join(pairs[1][0])
        reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
        reldict['objclass'] = pairs[1][1].label()
        reldict['objtext'] = _join(pairs[1][1].leaves())
        reldict['objsym'] = list2sym(pairs[1][1].leaves())
        reldict['rcon'] = _join(pairs[2][0][:window])
        if trace:
            print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
        result.append(reldict)
        pairs = pairs[1:]
    return result

semi_rel2reldict()的第一件事是检查tree2semi_rel()的输出中有多于2个元素，您的示例语句中没有:

The first thing that semi_rel2reldict() does is to check where there are more than 2 elements the output from tree2semi_rel(), which your example sentence doesn't:

>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> len(tree2semi_rel(chunked))
2
>>> len(tree2semi_rel(chunked)) > 2
False

啊哈，这就是extract_rel什么都不返回的原因.

Ah ha, that's why the extract_rel is returning nothing.

现在出现了一个问题，即即使有tree2semi_rel()中的2个元素，如何使extract_rel()返回某些内容?这可能吗?

Now comes the question of how to make extract_rel() return something even with 2 elements from tree2semi_rel()? Is that even possible?

让我们尝试其他句子:

>>> text = "Tom is the cofounder of Microsoft and now he is the founder of Marcohard"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')]), ('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN'), Tree('PERSON', [('Marcohard', 'NNP')])])
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])], [[('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN')], Tree('PERSON', [('Marcohard', 'NNP')])]]
>>> len(tree2semi_rel(chunked)) > 2
True
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': 'and/CC now/RB he/PRP is/VBZ the/DT', 'subjtext': 'Tom/NNP'})]

但这仅确认extract_rel在tree2semi_rel返回成对的<时不能提取. 2.如果我们删除while len(pairs) > 2的条件会怎样?

But that only confirms that extract_rel can't extract when tree2semi_rel returns pairs of < 2. What happens if we remove that condition of while len(pairs) > 2?

我们为什么不能while len(pairs) > 1?

如果我们仔细看一下代码，就会看到填充命令的最后一行，

If we look closer into the code, we see the last line of populating the reldict, https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L169:

reldict['rcon'] = _join(pairs[2][0][:window])

它尝试访问pairs的第三个元素，如果pairs的长度为2，则会得到一个IndexError.

It tries to access a 3rd element of the pairs and if the length of the pairs is 2, you'll get an IndexError.

那么，如果我们删除该rcon键并将其简单地更改为while len(pairs) >= 2，会发生什么?

So what happens if we remove that rcon key and simply change it to while len(pairs) >= 2?

为此，我们必须覆盖semi_rel2redict()函数:

To do that we have to override the semi_rel2redict() function:

>>> from nltk.sem.relextract import _join, list2sym
>>> from collections import defaultdict
>>> def semi_rel2reldict(pairs, window=5, trace=False):
...     """
...     Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
...     stores information about the subject and object NEs plus the filler between them.
...     Additionally, a left and right context of length =< window are captured (within
...     a given input sentence).
...     :param pairs: a pair of list(str) and ``Tree``, as generated by
...     :param window: a threshold for the number of items to include in the left and right context
...     :type window: int
...     :return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
...     :rtype: list(defaultdict)
...     """
...     result = []
...     while len(pairs) >= 2:
...         reldict = defaultdict(str)
...         reldict['lcon'] = _join(pairs[0][0][-window:])
...         reldict['subjclass'] = pairs[0][1].label()
...         reldict['subjtext'] = _join(pairs[0][1].leaves())
...         reldict['subjsym'] = list2sym(pairs[0][1].leaves())
...         reldict['filler'] = _join(pairs[1][0])
...         reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
...         reldict['objclass'] = pairs[1][1].label()
...         reldict['objtext'] = _join(pairs[1][1].leaves())
...         reldict['objsym'] = list2sym(pairs[1][1].leaves())
...         reldict['rcon'] = []
...         if trace:
...             print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
...         result.append(reldict)
...         pairs = pairs[1:]
...     return result
... 
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]

啊！可以，但是extract_rels()仍然有第4步.

Ah! It works but there's still a 4th step in extract_rels().

relfilter = lambda x: (x['subjclass'] == subjclass and
                       len(x['filler'].split()) <= window and
                       pattern.match(x['filler']) and
                       x['objclass'] == objclass)

现在，让我们尝试使用被黑客入侵的semi_rel2reldict版本:

Now let's try it with the hacked version of semi_rel2reldict:

>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]
>>> 
>>> pattern = re.compile(r'.*\bof\b.*')
>>> reldicts = semi_rel2reldict(tree2semi_rel(chunked))
>>> relfilter = lambda x: (x['subjclass'] == subjclass and
...                            len(x['filler'].split()) <= window and
...                            pattern.match(x['filler']) and
...                            x['objclass'] == objclass)
>>> relfilter
<function <lambda> at 0x112e591b8>
>>> subjclass = 'PERSON'
>>> objclass = 'ORGANIZATION'
>>> window = 5
>>> list(filter(relfilter, reldicts))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]

有效！现在让我们以元组形式查看它:

It works! Now let's see it in tuple form:

>>> from nltk.sem.relextract import rtuple
>>> rels = list(filter(relfilter, reldicts))
>>> for rel in rels:
...     print rtuple(rel)
... 
[PER: 'Tom/NNP'] 'is/VBZ the/DT cofounder/NN of/IN' [ORG: 'Microsoft/NNP']

这篇关于NLTK关系提取不返回任何内容的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！