语义词法谓词性能 [英] Semantic lexer predicate performance

问题描述

我有一个词法分析器为传递给词法分析器的宏字符串的动态列表创建MACRO令牌.我在最高级的词法分析器规则中使用了语义谓词来实现此功能:

I have a lexer creates MACRO tokens for a dynamic list of macro strings passed to the lexer. I used a semantic predicate in the very top lexer rule to implement this feature:

MACRO: { macros != null && tryMacro() }? .;

tryMacro()仅检查是否有任何宏字符串与输入序列匹配.

Where tryMacro() just checks if any macro string matches the input sequence.

这种方法的性能非常差，经过一些研究，我尝试将词法分析器规则更改为以下内容:

The performance of this approach was very bad and after some research I tried changing the lexer rule to the following:

MACRO: . { macros != null && tryMacro() }?;

这严重改善了性能，但我真的不明白为什么. :)自'.'匹配任何字符，语义谓词规则应该被完全调用与以前一样的次数，不是吗?有人可以提供这种行为的解释吗?

This severely improved the performance but I don't really understand why. :) Since the '.' matches any character, the semantic predicate rule should be invoked exactly as many times as before, shouldn't it? Can someone provide an explanation for this behavior?

推荐答案

我对此进行了进一步调试，结果发现规则的重新排序改变了词法分析器的行为，从而导致在解析过程中宏不被接受.感觉到的性能提高的原因是因为语义谓词仅在词法分析器在进行预测时放弃该规则之前进行了两次评估.因此，规则的更改实际上是无效的，而不是性能的提高.

I debugged this a bit further and it turned out that the reordering of the rule changed the behavior of the lexer causing macros to not be accepted during parsing. The reason for the perceived increase in performance was because the semantic predicate was only evaluated a couple of times before the lexer dropped the rule while doing its predictions. So the change of the rule was actually invalid and not a performance improvement.

我终于通过将宏处理移至解析器解决了性能问题.

I finally solved the performance issue by moving the macro handling to the parser.

这篇关于语义词法谓词性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！