动态创建词法分析器规则 [英] Dynamically create lexer rule

问题描述

这是一个简单的规则:

NAME : 'name1' | 'name2' | 'name3';

是否可以使用包含字符串的数组为此类规则动态提供替代方法?

Is it possible to provide alternatives for such rule dynamically using an array that contains strings?

推荐答案

是的，动态令牌与IDENTIFIER规则匹配

Yes, dynamic tokens match IDENTIFIER rule

在那种情况下，只需在Id完全匹配之后进行 检查，以查看Id匹配的文本是否在预定义的集合中.如果它在集合中(在我的示例中为Set)，请更改令牌的类型.

In that case, simply do a check after the Id has matched completely to see if the text the Id matched is in a predefined collection. If it is in the collection (a Set in my example) change the type of the token.

一个小演示:

grammar T;

@lexer::members {
  private java.util.Set<String> special;

  public TLexer(ANTLRStringStream input, java.util.Set<String> special) {
    super(input);
    this.special = special;
  }

}

parse
 : (t=. {System.out.printf("\%-10s'\%s'\n", tokenNames[$t.type], $t.text);})* EOF
 ;

Id
 : ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '0'..'9')*
   {if(special.contains($text)) $type=Special;}
 ;

Int
 : '0'..'9'+
 ;

Space
 : (' ' | '\t' | '\r' | '\n') {skip();}
 ;

fragment Special : ;

如果您现在运行以下演示，则:

And if you now run the following demo:

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    String source = "foo bar baz Mu";
    java.util.Set<String> set = new java.util.HashSet<String>();
    set.add("Mu");
    set.add("bar");
    TLexer lexer = new TLexer(new ANTLRStringStream(source), set);
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.parse();
  }
}

您将看到以下打印内容:

You will see the following being printed:

Id        'foo'
Special   'bar'
Id        'baz'
Special   'Mu'

ANTLR4

对于ANTLR4，您可以执行以下操作:

ANTLR4

For ANTLR4, you can do something like this:

grammar T;

@lexer::members {
  private java.util.Set<String> special = new java.util.HashSet<>();

  public TLexer(CharStream input, java.util.Set<String> special) {
    this(input);
    this.special = special;
  }
}

tokens {
  Special
}

parse
 : .*? EOF
 ;

Id
 : [a-zA-Z_] [a-zA-Z_0-9]* {if(special.contains(getText())) setType(TParser.Special);}
 ;

Int
 : [0-9]+
 ;

Space
 : [ \t\r\n] -> skip
 ;

通过课程进行测试:

import org.antlr.v4.runtime.*;
import java.util.HashSet;
import java.util.Set;

public class Main {

  public static void main(String[] args) {

    String source = "foo bar baz Mu";
    Set<String> set = new HashSet<String>(){{
      add("Mu");
      add("bar");
    }};

    TLexer lexer = new TLexer(CharStreams.fromString(source), set);
    CommonTokenStream tokenStream = new CommonTokenStream(lexer);
    tokenStream.fill();

    for (Token t : tokenStream.getTokens()) {
      System.out.printf("%-10s '%s'\n", TParser.VOCABULARY.getSymbolicName(t.getType()), t.getText());
    }
  }
}

将打印:

Id         'foo'
Special    'bar'
Id         'baz'
Special    'Mu'
EOF        '<EOF>'

这篇关于动态创建词法分析器规则的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！