编写对空白敏感的解析器规则，同时从词法分析器中跳过WS [英] Writing parser rules sensitive to whitespace while skipping WS from the lexer

问题描述

我在处理空白时遇到了一些麻烦.在下面的语法摘录中，我设置了词法分析器，以便解析器跳过空白:

I am having some troubles in handling whitespace. In the following excerpt of a grammar, I set up the lexer so that the parser skips whitespace:

ENTITY_VAR
    : 'user'
    | 'resource'
    ;

INT : DIGIT+ | '-' DIGIT+ ;
ID : LETTER (LETTER | DIGIT | SPECIAL)* ;
ENTITY_ID : '__' ENTITY_VAR ('_w_' ID)?;

NEWLINE : '\r'? '\n';

WS : [ \t\r\n]+ -> skip; // skip spaces, tabs, newlines

fragment LETTER : [a-zA-Z];
fragment DIGIT : [0-9];
fragment SPECIAL : ('_' | '#' );

问题是，我想与形式为ENTITY_ID的变量名称匹配，以使匹配的字符串没有任何空格.像我在这里一样将其编写为词法分析器规则就足够了，但事实是我想使用解析器规则来代替它，因为我想直接访问这两个标记ENTITY_VAR和ID从我的代码中单独获取，而不是将它们重新压缩成一个完整的令牌ENTITY_ID.

The problem is, I would like to match against variables names of the form ENTITY_ID such that the matched string does not have any whitespace. It would be sufficient to write it as a lexer rule as I did here, but the thing is that I'd like to do it with a parser rule instead, because I want to have direct access to those two tokens ENTITY_VAR and ID individually from my code, and not squeeze them back together in a whole token ENTITY_ID.

有什么想法吗? 基本上，任何允许我直接访问ENTITY_VAR和ID的解决方案都适合我，只需将ENTITY_ID保留为词法分析器规则或将其移至解析器即可.

Any ideas, please? Basically any solution which let me access directly ENTITY_VAR and ID would suit me, both by leaving ENTITY_ID as a lexer rule or moving it to the parser.

推荐答案

我可以想到几种方法(不是按特殊顺序排列):

There are several approaches I can think of (not in a special order):

1. 从规则ENTITY_ID发出多个令牌.请参见 ANTLR4:如何注入令牌以获取启发
2. 在解析器中允许空白，然后再检查
3. 使用单个令牌并拆分代码
4. 使用单个令牌并在将令牌流传递给解析器之前修改令牌流. IE. lex，修改ENTITY_ID令牌并将其拆分为其他几个令牌，然后将此流传递给解析器
5. 不要跳过空格，当处理这些额外令牌"时，请检查它们是否在ENTITY_ID部分内(=>是错误的)或否(=>忽略错误).
6. 请勿跳过空格，并在语法中允许空格的所有位置添加"WS *"(如果语法不太大，则可以).
7. 在分析器规则中插入谓词，以检查两者之间是否存在空格.

8. 创建一个陷阱"规则，如下所示:

1. Emit several tokens from the rule ENTITY_ID. See ANTLR4: How to inject tokens for an inspiration
2. Allow whitespace in the parser and check afterwards
3. Use the single token and split in code
4. Use the single token and modify the token stream before passing it to the parser. I.e. lex, modify the ENTITY_ID tokens and split them into several other tokens, then pass this stream to the parser
5. Don't skip whitespace and when dealing with these "extra tokens" check if they are within a ENTITY_ID part (=> is error) or not (=> ignore error).
6. Don't skip whitespace and add "WS*" everywhere in your grammar where whitespace is allowed (ok if the grammar is not too large).
7. Insert predicates in the parser rule that checks if there is whitespace between.

8. Create a "trap" rule like this:

INVALID_ENTITY_ID : '__' WS+ ENTITY_VAR WS? ('_w_' WS? ID)?
                  | '__' WS? ENTITY_VAR WS+ ('_w_' WS? ID)?
                  | '__' WS? ENTITY_VAR WS? ('_w_' WS+ ID)
                  ;

这将捕获无效的ENTITY_ID，因为它的长度大于将成为单独令牌的部分.

This will catch invalid ENTITY_IDs since it's longer than the parts that will then be also individual tokens.

如果不是在非错误"情况下不会更改解析，我将选择2，即通过允许空格来解释不同的代码.

I'd go with 2, if it doesn't alter the parse in the "non error" case, i.e. no code is interpreted differently by allowing whitespace.

这篇关于编写对空白敏感的解析器规则，同时从词法分析器中跳过WS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！