ANTLR 词法分析器根本无法向前看 [英] ANTLR lexer can't lookahead at all

问题描述

我有以下语法:

rule: 'aaa' | 'a' 'a';

它可以成功解析字符串'aaa'，但是解析'aa'失败，报错如下:

It can successfully parse the string 'aaa', but it fails to parse 'aa' with the following error:

line 1:2 mismatched character '<EOF>' expecting 'a'

仅供参考，这是词法分析器的问题而不是解析器的问题，因为我什至不调用解析器.主要功能如下:

FYI, it is the lexer's problem not the parser's because I don't even call the parser. The main function looks like:

@members {
  public static void main(String[] args) throws Exception {
    RecipeLexer lexer = new RecipeLexer(new ANTLRInputStream(System.in));
    for (Token t = lexer.nextToken(); t.getType() != EOF; t = lexer.nextToken())
      System.out.println(t.getType());
  }
}

结果与更明显的版本相同:

The result is the same with the more obvious version:

rule: AAA | A A;
AAA: 'aaa';
A: 'a';

显然，ANTLR 词法分析器试图将输入aa"与失败的规则 AAA 匹配.除了 ANTLR 是一个 LL(*) 解析器或其他什么之外，词法分析器应该与解析器分开工作，并且应该能够解决歧义.语法与旧的 lex(或 flex)一起工作得很好，但它似乎不适用于 ANTLR.那么这里的问题是什么?

Obviously the ANTLR lexer tries to match the input 'aa' with the rule AAA which fails. Apart from that ANTLR is an LL(*) parser or whatever, the lexer should work separately from the parser and it should be able to resolve ambiguity. The grammar works fine with the good old lex(or flex) but it doesn't seem with ANTLR. So what is the problem here?

感谢您的帮助！

推荐答案

ANTLR 生成的解析器是(或可以是)LL(*)，而不是它的词法分析器.

ANTLR's generated parsers are (or can be) LL(*), not its lexers.

当词法分析器看到输入的"aa"时，它会尝试匹配标记AAA.当它失败时，它会尝试匹配也匹配 "aa" 的任何其他标记(词法分析器不会回溯以匹配 A！).由于这是不可能的，因此产生了错误.

When the lexer sees the input "aa", it tries to match token AAA. When it fails to do so, it tries to match any other token that also matches "aa" (the lexer does not backtrack to match A!). Since this is not possible, an error is produced.

这通常不是问题，因为在实践中，通常可以使用某种标识符规则 "aa" .那么，您是想解决什么实际问题，还是只是对内部运作感到好奇?如果是第一个，请编辑您的问题并描述您的实际问题.

This is usually not a problem, since in practice, there's often some sort of identifier rule "aa" can fall back to. So, what actual problem are you trying solve, or were you only curious of the inner workings? If it's the first, please edit your question and describe your actual problem.

这篇关于ANTLR 词法分析器根本无法向前看的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！