ANTLR 是否可以使用嵌入语法来制作语法? [英] ANTLR Is it possible to make grammar with embed grammar inside?

问题描述

ANTLR:是否可以在内部使用嵌入语法(使用它自己的词法分析器)制作语法?

ANTLR: Is it possible to make grammar with embed grammar (with it's own lexer) inside?

例如在我的语言中，我有能力使用嵌入 SQL 语言:

For example in my language I have ability to use embed SQL language:

var Query = [select * from table];
with Query do something ....;

ANTLR 可以吗?

推荐答案

是否可以在内部使用嵌入语法(使用它自己的词法分析器)制作语法?

Is it possible to make grammar with embed grammar (with it's own lexer) inside?

如果您的意思是是否可以在一个语法中定义两种语言(使用单独的词法分析器)，那么答案是:不，那是不可能的.

If you mean whether it is possible to define two languages in a single grammar (using separate lexers), then the answer is: no, that's not possible.

然而，如果问题是是否有可能将两种语言解析成一个单一的 AST，那么答案是:是的，有可能.

However, if the question is whether it is possible to parse two languages into a single AST, then the answer is: yes, it is possible.

您只需要:

• 用自己的语法定义两种语言；
• 在您的主要语法中创建一个词法分析器规则，以捕获嵌入式语言的整个输入；
• 使用重写规则调用自定义方法，该方法解析外部 AST 并使用 { ... } 将其插入到主 AST 中(请参阅 expr 规则(MyLanguage.g)).

• define both languages in their own grammar;
• create a lexer rule in you main grammar that captures the entire input of the embedded language;
• use a rewrite rule that calls a custom method that parses the external AST and inserts it in the main AST using { ... } (see the expr rule in the main grammar (MyLanguage.g)).

grammar MyLanguage;

options {
  output=AST;
  ASTLabelType=CommonTree;
}

tokens {
  ROOT;
}

@members {
  private CommonTree parseSQL(String sqlSrc) {
    try {
      MiniSQLLexer lexer = new MiniSQLLexer(new ANTLRStringStream(sqlSrc));
      MiniSQLParser parser = new MiniSQLParser(new CommonTokenStream(lexer));
      return (CommonTree)parser.parse().getTree();
    } catch(Exception e) {
      return new CommonTree(new CommonToken(-1, e.getMessage()));
    }
  }
}

parse
  :  assignment+ EOF -> ^(ROOT assignment+)
  ;

assignment
  :  Var Id '=' expr ';' -> ^('=' Id expr)
  ;

expr
  :  Num
  |  SQL -> {parseSQL($SQL.text)}
  ;

Var   : 'var';
Id    : ('a'..'z' | 'A'..'Z')+;
Num   : '0'..'9'+;
SQL   : '[' ~']'* ']';
Space : ' ' {skip();};

MiniSQL.g

grammar MiniSQL;

options {
  output=AST;
  ASTLabelType=CommonTree;
}

parse
  :  '[' statement ']' EOF -> statement
  ;

statement
  :  select
  ;

select
  :  Select '*' From ID -> ^(Select '*' From ID)
  ;

Select : 'select';
From   : 'from';
ID     : ('a'..'z' | 'A'..'Z')+;
Space  : ' ' {skip();};

Main.java

import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;

public class Main {
  public static void main(String[] args) throws Exception {
    String src = "var Query = [select * from table]; var x = 42;";
    MyLanguageLexer lexer = new MyLanguageLexer(new ANTLRStringStream(src));
    MyLanguageParser parser = new MyLanguageParser(new CommonTokenStream(lexer));
    CommonTree tree = (CommonTree)parser.parse().getTree();
    DOTTreeGenerator gen = new DOTTreeGenerator();
    StringTemplate st = gen.toDOT(tree);
    System.out.println(st);
  }
}

运行演示

java -cp antlr-3.3.jar org.antlr.Tool MiniSQL.g 
java -cp antlr-3.3.jar org.antlr.Tool MyLanguage.g 
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main

给定输入:

var Query = [select * from table]; var x = 42;

Main 类的输出对应于以下 AST:

the output of the Main class corresponds to the following AST:

如果您想在 SQL 中允许字符串文字(可能包含 ])和注释(可能包含 ' 和 ])，您可以在主要语法中使用以下 SQL 规则:

And if you want to allow string literals inside your SQL (which could contain ]), and comments (which could contain ' and ]), the you could use the following SQL rule inside your main grammar:

SQL
  :  '[' ( ~(']' | '\'' | '-')
         | '-' ~'-' 
         | COMMENT 
         | STR
         )* 
     ']'
  ;

fragment STR 
  :  '\'' (~('\'' | '\r' | '\n') | '\'\'')+ '\'' 
  |  '\'\''
  ;

fragment COMMENT
  :  '--' ~('\r' | '\n')*
  ;

这将正确解析单个标记中的以下输入:

which would properly parse the following input in a single token:

[
  select a,b,c 
  from table 
  where a='A''B]C' 
  and b='' -- some ] comment ] here'
]

请注意，尝试为整个 SQL 方言(甚至是大子集)创建语法并非易事！您可能想要搜索现有的 SQL 解析器，或查看 ANTLR wiki 以获取 example-grammars.

Just beware that trying to create a grammar for an entire SQL dialect (or even a large subset) is no trivial task! You may want to search for existing SQL parsers, or look at the ANTLR wiki for example-grammars.

这篇关于ANTLR 是否可以使用嵌入语法来制作语法?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！