使用 spaCy 3 的自定义 NER 训练抛出 ValueError [英] Custom NERs training with spaCy 3 throws ValueError

问题描述

我正在尝试使用 spacy 3 添加自定义 NER 标签.我找到了旧版本的教程并对 spacy 3 进行了调整.这是我正在使用的整个代码:

I am trying to add custom NER labels using spacy 3. I found tutorials for older versions and made adjustments for spacy 3. Here is the whole code I am using:

import random
import spacy
from spacy.training import Example

LABEL = 'ANIMAL'
TRAIN_DATA = [
    ("Horses are too tall and they pretend to care about your feelings", {'entities': [(0, 6, LABEL)]}),
    ("Do they bite?", {'entities': []}),
    ("horses are too tall and they pretend to care about your feelings", {'entities': [(0, 6, LABEL)]}),
    ("horses pretend to care about your feelings", {'entities': [(0, 6, LABEL)]}),
    ("they pretend to care about your feelings, those horses", {'entities': [(48, 54, LABEL)]}),
    ("horses?", {'entities': [(0, 6, LABEL)]})
]
nlp = spacy.load('en_core_web_sm')  # load existing spaCy model
ner = nlp.get_pipe('ner')
ner.add_label(LABEL)
print(ner.move_names) # Here I see, that the new label was added
optimizer = nlp.create_optimizer()
# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
with nlp.disable_pipes(*other_pipes):  # only train NER
    for itn in range(20):
        random.shuffle(TRAIN_DATA)
        losses = {}
        for text, annotations in TRAIN_DATA:
            doc = nlp(text)
            example = Example.from_dict(doc, annotations)
            nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)
        print(losses)
# test the trained model # add some dummy sentences with many NERs

test_text = 'Do you like horses?'
doc = nlp(test_text)
print("Entities in '%s'" % test_text)
for ent in doc.ents:
    print(ent.label_, " -- ", ent.text)

此代码输出 ValueError 异常，但仅在 2 次迭代之后 - 注意前两行:

This code outputs the ValueError exception, but only after 2 iterations - notice the first 2 lines:

{'ner': 9.862242701536594}
{'ner': 8.169456698315201}
Traceback (most recent call last):
  File ".\custom_ner_training.py", line 46, in <module>
    nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)
  File "C:\ogr\moje\python\spacy_pg\myvenv\lib\site-packages\spacy\language.py", line 1106, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "spacy\pipeline\transition_parser.pyx", line 366, in spacy.pipeline.transition_parser.Parser.update
  File "spacy\pipeline\transition_parser.pyx", line 478, in spacy.pipeline.transition_parser.Parser.get_batch_loss
  File "spacy\pipeline\_parser_internals\ner.pyx", line 310, in spacy.pipeline._parser_internals.ner.BiluoPushDown.set_costs
ValueError

我看到 ANIMAL 标签是通过调用 ner.move_names 添加的.

I see the ANIMAL label was added by calling ner.move_names.

当我更改值 LABEL = 'PERSON 时，代码成功运行并在新数据上将马识别为 PERSON.这就是为什么我假设代码本身没有错误.

When I change my the value LABEL = 'PERSON, the code runs successfully and recognizes horses as PERSON on the new data. This is why I am assuming, there is no error in the code itself.

有什么我遗漏的吗?我究竟做错了什么?请问有人可以复制吗?

Is there something I am missing? What am I doing wrong? Could someone reproduce, please?

注意:这是我在这里的第一个问题.我希望我提供了所有信息.如果没有，请在评论中告诉我.

NOTE: This is my first question ever here. I hope I provided all information. If not, let me know in the comments.

推荐答案

for 循环中需要更改以下行

You need to change the following line in the for loop

doc = nlp(text)

到

doc = nlp.make_doc(text)

代码应该可以工作并产生以下结果:

The code should work and produce the following results:

{'ner': 9.60289144264557}
{'ner': 8.875474230820478}
{'ner': 6.370401408220459}
{'ner': 6.687456469517201}
... 
{'ner': 1.3796682589133492e-05}
{'ner': 1.7709562613218738e-05}

Entities in 'Do you like horses?'
ANIMAL  --  horses

这篇关于使用 spaCy 3 的自定义 NER 训练抛出 ValueError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！