在antlr4 lexer中，如何具有捕获所有剩余“单词"的规则?作为未知令牌? [英] In antlr4 lexer, How to have a rule that catches all remaining "words" as Unknown token?

问题描述

我有一个antlr4词法分析器语法.它有许多单词规则，但我也希望它为其他规则无法匹配的任何单词创建一个未知令牌.我有这样的东西:

I have an antlr4 lexer grammar. It has many rules for words, but I also want it to create an Unknown token for any word that it can not match by other rules. I have something like this:

Whitespace : [ \t\n\r]+ -> skip;
Punctuation : [.,:;?!];
// Other rules here
Unknown : .+? ; 

现在生成的匹配器将'〜'捕获为未知，但为输入'~~~'创建3个'〜'未知令牌，而不是单个'~~~'令牌.我该怎么做才能告诉lexer为未知的连续字符生成单词令牌.我也尝试过未知:.；"和未知:.+;"没有结果.

Now generated matcher catches '~' as unknown but creates 3 '~' Unknown tokens for input '~~~' instead of a single '~~~' token. What should I do to tell lexer to generate word tokens for unknown consecutive characters. I also tried "Unknown: . ;" and "Unknown : .+ ;" with no results.

编辑:在当前的antlr版本中.+?现在捕获了剩余的单词，因此似乎可以解决此问题.

In current antlr versions .+? now catches remaining words, so this problem seems to be resolved.

推荐答案

.+?始终匹配单个字符.但是.+将消耗尽可能多的能量，这在ANTLR v3(可能也是v4)的规则结尾处是非法的.

.+? at the end of a lexer rule will always match a single character. But .+ will consume as much as possible, which was illegal at the end of a rule in ANTLR v3 (v4 probably as well).

您可以做的只是匹配一个字符，然后在解析器中将它们粘合"在一起:

What you can do is just match a single char, and "glue" these together in the parser:

unknowns : Unknown+ ; 

...

Unknown  : . ; 

编辑

...但是我只有一个词法分析器，没有解析器...

... but I only have a lexer, no parsers ...

啊，我明白了.然后，您可以覆盖nextToken()方法:

Ah, I see. Then you could override the nextToken() method:

lexer grammar Lex;

@members {

  public static void main(String[] args) {
    Lex lex = new Lex(new ANTLRInputStream("foo, bar...\n"));
    for(Token t : lex.getAllTokens()) {
      System.out.printf("%-15s '%s'\n", tokenNames[t.getType()], t.getText());
    }
  }

  private java.util.Queue<Token> queue = new java.util.LinkedList<Token>();

  @Override
  public Token nextToken() {    

    if(!queue.isEmpty()) {
      return queue.poll();
    }

    Token next = super.nextToken();

    if(next.getType() != Unknown) {
      return next;
    }

    StringBuilder builder = new StringBuilder();

    while(next.getType() == Unknown) {
      builder.append(next.getText());
      next = super.nextToken();
    }

    // The `next` will _not_ be an Unknown-token, store it in 
    // the queue to return the next time!
    queue.offer(next);

    return new CommonToken(Unknown, builder.toString());
  }
}

Whitespace  : [ \t\n\r]+ -> skip ;
Punctuation : [.,:;?!] ;
Unknown     : . ; 

运行它:

java -cp antlr-4.0-complete.jar org.antlr.v4.Tool Lex.g4 
javac -cp antlr-4.0-complete.jar *.java
java -cp .:antlr-4.0-complete.jar Lex

将打印:

Unknown         'foo'
Punctuation     ','
Unknown         'bar'
Punctuation     '.'
Punctuation     '.'
Punctuation     '.'

这篇关于在antlr4 lexer中，如何具有捕获所有剩余“单词"的规则?作为未知令牌?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！