ANTLR4 在 Python 中生成解析器很慢，但在 Java 中很快 [英] Slow ANTLR4 generated Parser in Python, but fast in Java

问题描述

我正在尝试将 ant 读到python可能比java慢20-30倍，但在我的情况下，python 慢了大约 400 倍！

解决方案

我确认 Python 2 和 Python 3 运行时存在性能问题.通过几个补丁，我在 python3 运行时获得了 10 倍的加速(约 5 秒到约 400 毫秒).https://github.com/antlr/antlr4/pull/1010

I am trying to convert ant ANTLR3 grammar to an ANTLR4 grammar, in order to use it with the antlr4-python2-runtime. This grammar is a C/C++ fuzzy parser.

After converting it (basically removing tree operators and semantic/syntactic predicates), I generated the Python2 files using:

java -jar antlr4.5-complete.jar -Dlanguage=Python2 CPPGrammar.g4

And the code is generated without any error, so I import it in my python project (I'm using PyCharm) to make some tests:

import sys, time
from antlr4 import *
from parser.CPPGrammarLexer import CPPGrammarLexer
from parser.CPPGrammarParser import CPPGrammarParser

currenttimemillis = lambda: int(round(time.time() * 1000))

def is_string(object):
    return isinstance(object,str)

def parsecommandstringline(argv):
    if(2!=len(argv)):
        raise IndexError("Invalid args size.")
    if(is_string(argv[1])):
        return True
    else:
        raise TypeError("Argument must be str type.")

def doparsing(argv):
    if parsecommandstringline(argv):
        print("Arguments: OK - {0}".format(argv[1]))
        input = FileStream(argv[1])
        lexer = CPPGrammarLexer(input)
        stream = CommonTokenStream(lexer)
        parser = CPPGrammarParser(stream)
        print("*** Parser: START ***")
        start = currenttimemillis()
        tree = parser.code()
        print("*** Parser: END *** - {0} ms.".format(currenttimemillis()-start))
        pass

def main(argv):
    tree = doparsing(argv)
    pass

if __name__ == '__main__':
    main(sys.argv)

The problem is that the parsing is very slow. With a file containing ~200 lines it takes more than 5 minutes to complete, while the parsing of the same file in antlrworks only takes 1-2 seconds. Analyzing the antlrworks tree, I noticed that the expr rule and all of its descendants are called very often and I think that I need to simplify/change these rules to make the parser operate faster:

Is my assumption correct or did I make some mistake while converting the grammar? What can be done to make parsing as fast as on antlrworks?

UPDATE: I exported the same grammar to Java and it only took 795ms to complete the parsing. The problem seems more related to python implementation than to the grammar itself. Is there anything that can be done to speed up Python parsing?
I've read here that python can be 20-30 times slower than java, but in my case python is ~400 times slower!

解决方案

I confirm that the Python 2 and Python 3 runtimes have performance issues. With a few patches, I got a 10x speedup on the python3 runtime (~5 seconds down to ~400 ms). https://github.com/antlr/antlr4/pull/1010

这篇关于ANTLR4 在 Python 中生成解析器很慢，但在 Java 中很快的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！