ANTLR4:隐式或显式令牌定义 [英] ANTLR4: implicit or explicit token definition

问题描述

在ANTLR4中使用显式令牌定义的优点和缺点是什么?与创建单独的标记并代替文本使用令牌相比，我发现在单个括号中的文本更具描述性，更易于使用.

What are the benefits and drawbacks of using explicit token definitions in ANTLR4? I find the text in single parentheses more descriptive and easier to use than creating a separate token and using that in place of the text.

例如:

grammar SimpleTest;

top: library | module ;

library: 'library' library_name ';' ;
library_name: IDENTIFIER;         

module: MODULE module_name ';' ;
module_name: IDENTIFIER;

MODULE: 'module' ;
IDENTIFIER: [a-zA-Z0-9]+;

生成的令牌为:

T__0=1
T__1=2
MODULE=3
IDENTIFIER=4
'library'=1
';'=2
'module'=3

如果我对'library'令牌"不感兴趣，因为该规则已经确定了我要匹配的对象，并且无论如何我都将跳过它，用LIBRARY替换它是否有意义?令牌声明? (然后令牌的数量将增加.)为什么这在ANTLRWorks中是警告?

If I am not interested in the 'library' "token", since the rule already establishes what I am matching against, and I will just skip over it anyway, does it make any sense to replace it with LIBRARY and a token declaration? (The number of tokens then will grow.) Why is this a warning in ANTLRWorks?

推荐答案

实际上，隐式和显式标记之间是有区别的:

Actually, there is a difference between implicit and explicit tokens:

摘自《权威的ANTLR4参考》，第76页:

From "The Definitive ANTLR4 Reference", page 76:

ANTLR收集并分离所有字符串文字和词法分析器 解析器规则中的规则.诸如枚举"之类的文字成为词汇 规则并立即在解析器规则之后但在 明确的词汇规则.

ANTLR collects and separates all of the string literals and lexer rules from the parser rules. Literals such as 'enum' become lexical rules and go immediately after the parser rules but before the explicit lexical rules.

ANTLR词法分析器解决了之间的歧义 通过支持首先指定的规则来建立词汇规则.

ANTLR lexers resolve ambiguities between lexical rules by favoring the rule specified first.

我的亮点.

这篇关于ANTLR4:隐式或显式令牌定义的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！