ANTLR lexer根本无法超前 [英] ANTLR lexer can't lookahead at all

问题描述

我有以下语法:

rule: 'aaa' | 'a' 'a';

它可以成功解析字符串'aaa'，但无法解析'aa'并显示以下错误:

It can successfully parse the string 'aaa', but it fails to parse 'aa' with the following error:

line 1:2 mismatched character '<EOF>' expecting 'a'

仅供参考，这是词法分析器的问题，而不是解析器的问题，因为我什至不调用解析器.主要功能如下:

FYI, it is the lexer's problem not the parser's because I don't even call the parser. The main function looks like:

@members {
  public static void main(String[] args) throws Exception {
    RecipeLexer lexer = new RecipeLexer(new ANTLRInputStream(System.in));
    for (Token t = lexer.nextToken(); t.getType() != EOF; t = lexer.nextToken())
      System.out.println(t.getType());
  }
}

结果与更明显的版本相同:

The result is the same with the more obvious version:

rule: AAA | A A;
AAA: 'aaa';
A: 'a';

很显然，ANTLR词法分析器尝试将输入"aa"与失败的规则AAA匹配.除了ANTLR是LL(*)解析器或其他解析器之外，词法分析器应该与解析器分开工作，并且它应该能够解决歧义.语法可以与旧的 lex (或 flex )一起使用，但对于ANTLR来说似乎并不可行.那么，这是什么问题呢?

Obviously the ANTLR lexer tries to match the input 'aa' with the rule AAA which fails. Apart from that ANTLR is an LL(*) parser or whatever, the lexer should work separately from the parser and it should be able to resolve ambiguity. The grammar works fine with the good old lex(or flex) but it doesn't seem with ANTLR. So what is the problem here?

感谢您的帮助！

推荐答案

ANTLR生成的解析器是(或可以是)LL(*)，而不是其词法分析器.

ANTLR's generated parsers are (or can be) LL(*), not its lexers.

当词法分析器看到输入"aa"时，它将尝试匹配标记AAA.如果这样做失败，它将尝试匹配也匹配"aa"的任何其他令牌(词法分析器不会回溯匹配A！).由于这是不可能的，因此会产生错误.

When the lexer sees the input "aa", it tries to match token AAA. When it fails to do so, it tries to match any other token that also matches "aa" (the lexer does not backtrack to match A!). Since this is not possible, an error is produced.

这通常不是问题，因为在实践中，通常会有某种标识符规则"aa"可以使用.那么，您到底想解决什么实际问题，或者您只是对内部运作感到好奇?如果是第一次，请编辑您的问题并描述您的实际问题.

This is usually not a problem, since in practice, there's often some sort of identifier rule "aa" can fall back to. So, what actual problem are you trying solve, or were you only curious of the inner workings? If it's the first, please edit your question and describe your actual problem.

这篇关于ANTLR lexer根本无法超前的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！