ANTLR4:隐式或显式令牌定义 [英] ANTLR4: implicit or explicit token definition

问题描述

在 ANTLR4 中使用显式令牌定义的优点和缺点是什么?我发现单括号中的文本比创建单独的标记并使用它代替文本更具描述性和更易于使用.

What are the benefits and drawbacks of using explicit token definitions in ANTLR4? I find the text in single parentheses more descriptive and easier to use than creating a separate token and using that in place of the text.

例如:

grammar SimpleTest;

top: library | module ;

library: 'library' library_name ';' ;
library_name: IDENTIFIER;         

module: MODULE module_name ';' ;
module_name: IDENTIFIER;

MODULE: 'module' ;
IDENTIFIER: [a-zA-Z0-9]+;

生成的令牌是:

T__0=1
T__1=2
MODULE=3
IDENTIFIER=4
'library'=1
';'=2
'module'=3

如果我对 'library' token"不感兴趣，因为规则已经建立了我匹配的对象，无论如何我都会跳过它，这是否有意义用 LIBRARY 和令牌声明替换它?(然后令牌的数量会增加.)为什么这是 ANTLRWorks 中的警告?

If I am not interested in the 'library' "token", since the rule already establishes what I am matching against, and I will just skip over it anyway, does it make any sense to replace it with LIBRARY and a token declaration? (The number of tokens then will grow.) Why is this a warning in ANTLRWorks?

推荐答案

其实隐式令牌和显式令牌是有区别的:

Actually, there is a difference between implicit and explicit tokens:

来自权威的 ANTLR4 参考"，第 76 页:

From "The Definitive ANTLR4 Reference", page 76:

ANTLR 收集并分离所有字符串文字和词法分析器来自解析器规则的规则.像enum"这样的文字变成词法rules 并在解析器规则之后但在解析器规则之前明确的词汇规则.

ANTLR collects and separates all of the string literals and lexer rules from the parser rules. Literals such as 'enum' become lexical rules and go immediately after the parser rules but before the explicit lexical rules.

ANTLR 词法分析器解决了词法规则优先于指定的规则.

ANTLR lexers resolve ambiguities between lexical rules by favoring the rule specified first.

突出显示我.

这篇关于ANTLR4:隐式或显式令牌定义的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！