如何在Python中加速Stanford NLP? [英] How to speedup Stanford NLP in Python?

问题描述

import numpy as np
from nltk.tag import StanfordNERTagger
from nltk.tokenize import word_tokenize
    #english.all.3class.distsim.crf.ser.gz
st = StanfordNERTagger('/media/sf_codebase/modules/stanford-ner-2018-10-16/classifiers/english.all.3class.distsim.crf.ser.gz',
                           '/media/sf_codebase/modules/stanford-ner-2018-10-16/stanford-ner.jar',
                           encoding='utf-8')

初始化上面的代码Stanford NLP之后，下面的代码需要10秒钟来标记文本，如下所示.如何加快速度?

After initializing above code Stanford NLP following code takes 10 second to tag the text as shown below. How to speed up?

%%time
text="My name is John Doe"
tokenized_text = word_tokenize(text)
classified_text = st.tag(tokenized_text)
print (classified_text)

输出

[('My', 'O'), ('name', 'O'), ('is', 'O'), ('John', 'PERSON'), ('Doe', 'PERSON')]
CPU times: user 4 ms, sys: 20 ms, total: 24 ms
Wall time: 10.9 s

推荐答案

NLTK中的另一个解决方案是不使用旧的nltk.tag.StanfordNERTagger，而使用较新的nltk.parse.CoreNLPParser.参见例如 https://github.com/nltk/nltk/wiki/Stanford-CoreNLP-API-in-NLTK .

Another solution within NLTK is to not use the old nltk.tag.StanfordNERTagger but instead to use the newer nltk.parse.CoreNLPParser . See, e.g., https://github.com/nltk/nltk/wiki/Stanford-CoreNLP-API-in-NLTK .

通常，获得良好性能的秘诀确实是在Java端使用服务器，您可以重复调用该服务器而不必为处理的每个句子启动新的子流程.如果只需要NER，则可以使用NERServer或所有CoreNLP功能使用StanfordCoreNLPServer.有许多Python接口，请参见: https://stanfordnlp. github.io/CoreNLP/other-languages.html#python

More generally the secret to good performance is indeed to use a server on the Java side, which you can repeatedly call without having to start new subprocesses for each sentence processed. You can either use the NERServer if you just need NER or the StanfordCoreNLPServer for all CoreNLP functionality. There are a number of Python interfaces to it, see: https://stanfordnlp.github.io/CoreNLP/other-languages.html#python

这篇关于如何在Python中加速Stanford NLP?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！