ANTLR API问题;示例+解决方法提供;需要解释 [英] ANTLR API issue ; example + workaround provided ; explanation required

问题描述

我使用ANTLRWorks创建了以下Lexer。 （另请参见 http://bkiers.blogspot.com/ 2011/03 / 2-introduction-to-antlr.html #intro ）

I created the following Lexer using ANTLRWorks. ( See also http://bkiers.blogspot.com/2011/03/2-introduction-to-antlr.html#intro )

 // CSVLexer.g
 lexer grammar CSVLexer;

 @lexer::header {
   package graphica.parsers;
 }

 Comma   
   :  ','  
   ;  

 LineBreak  
  :  '\r'? '\n'  
  |  '\r'  
  ;  

 SimpleValue  
   :  ~(',' | '\r' | '\n' | '"')+  
   ;  

 QuotedValue  
   :  '"' ('""' | ~'"')* '"'  
   ;  

我使用以下Java类来测试Lexer。

I used the following Java class to test the Lexer.

 /**
  *
  * @author Nilo
  */
 import org.antlr.runtime.*;

 public class CSVLexerTest {

 public static void main(String[] args) throws Exception {
    // the input source  
    String source =
            "val1, value2, value3, value3.2" + "\n"
            + "\"line\nbreak\",ABAbb,end";

    // create an instance of the lexer  
    CSVLexer lexer = new CSVLexer(new ANTLRStringStream(source));
    // wrap a token-stream around the lexer  
    CommonTokenStream tokens = new CommonTokenStream(lexer);


    // traverse the tokens and print them to see if the correct tokens are created
    // tokens.toString();
    int n = 1;
    for (Object o : tokens.getTokens()) {
        CommonToken token = (CommonToken) o;
        System.out.println("token(" + n + ") = " + token.getText().replace("\n",    "\\n"));
        n++;
    }
 }
 }

上面的类（来自相同的教程）不产生任何输出。但是，如果我在令牌循环之前插入一个tokens.toString（），那么输出将按预期打印。

The class above ( from the same tutorial ) does NOT produce any output. If I however insert a tokens.toString() prior to the token loop then output is printed as expected.

注意：我使用的是ANTLWorks 1.4.3，ANTLR 3.4，带有JDK 1.7 / 64bit的Windows 7

Note: I use ANTLWorks 1.4.3, ANTLR 3.4, on Windows 7 with JDK 1.7/64bit

问题：我不明白这一点。请解释。如果没有tokens.toString（）

推荐答案

CommonTokenStream extends BufferedTokenStream ，其中包含 List< Token>当一个人调用 getTokens（）时返回的令牌。但是这个列表<令牌>令牌仅在特定时间填写。在3.3和3.4中，它 not 在 getTokens（）之后发生，其中3.2 填充令牌列表。

CommonTokenStream extends BufferedTokenStream which has a List<Token> tokens that is returned when one calls getTokens(). But this List<Token> tokens only gets filled at certain times. In 3.3 and 3.4 it does not happen after getTokens() where 3.2 does fill the tokens list.

public List getTokens() {
    if ( p == -1 ) {
        fillBuffer();
    }
    return tokens;
}

protected void fillBuffer() {
    // fill `tokens`
}

ANTLR 3.3（以及之后）

ANTLR 3.3 (and after)

public List getTokens() { 
    return tokens; 
}

public void fill() {
    // fill `tokens`
}

注意3.2的填充方法是如何受到保护的，在3.3+中它是公共的，所以以下是有效的：

Notice how 3.2's fill method is protected and in 3.3+ it is public, so the following works:

import org.antlr.runtime.*;

public class CSVLexerTest {

  public static void main(String[] args) throws Exception {

    // the input source  
    String source =
        "val1, value2, value3, value3.2" + "\n" + 
        "\"line\nbreak\",ABAbb,end";

    // create an instance of the lexer  
    CSVLexer lexer = new CSVLexer(new ANTLRStringStream(source));

    // wrap a token-stream around the lexer and fill the tokens-list 
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    tokens.fill();

    // traverse the tokens and print them to see if the correct tokens are created
    // tokens.toString();
    int n = 1;
    for (Object o : tokens.getTokens()) {
      CommonToken token = (CommonToken) o;
      System.out.println("token(" + n + ") = " + token.getText().replace("\n",    "\\n"));
      n++;
    }
  }
}

产生输出：

token(1) = val1
token(2) = ,
token(3) =  value2
token(4) = ,
token(5) =  value3
token(6) = ,
token(7) =  value3.2
token(8) = \n
token(9) = "line\nbreak"
token(10) = ,
token(11) = ABAbb
token(12) = ,
token(13) = end
token(14) = <EOF>

这篇关于ANTLR API问题;示例+解决方法提供;需要解释的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！