ANTLR API 问题；提供示例+解决方法；需要解释 [英] ANTLR API issue ; example + workaround provided ; explanation required

问题描述

我使用 ANTLRWorks 创建了以下词法分析器.(另见 http://bkiers.blogspot.com/2011/03/2-introduction-to-antlr.html#intro )

I created the following Lexer using ANTLRWorks. ( See also http://bkiers.blogspot.com/2011/03/2-introduction-to-antlr.html#intro )

 // CSVLexer.g
 lexer grammar CSVLexer;

 @lexer::header {
   package graphica.parsers;
 }

 Comma   
   :  ','  
   ;  

 LineBreak  
  :  '\r'? '\n'  
  |  '\r'  
  ;  

 SimpleValue  
   :  ~(',' | '\r' | '\n' | '"')+  
   ;  

 QuotedValue  
   :  '"' ('""' | ~'"')* '"'  
   ;  

我使用以下 Java 类来测试 Lexer.

I used the following Java class to test the Lexer.

 /**
  *
  * @author Nilo
  */
 import org.antlr.runtime.*;

 public class CSVLexerTest {

 public static void main(String[] args) throws Exception {
    // the input source  
    String source =
            "val1, value2, value3, value3.2" + "\n"
            + "\"line\nbreak\",ABAbb,end";

    // create an instance of the lexer  
    CSVLexer lexer = new CSVLexer(new ANTLRStringStream(source));
    // wrap a token-stream around the lexer  
    CommonTokenStream tokens = new CommonTokenStream(lexer);


    // traverse the tokens and print them to see if the correct tokens are created
    // tokens.toString();
    int n = 1;
    for (Object o : tokens.getTokens()) {
        CommonToken token = (CommonToken) o;
        System.out.println("token(" + n + ") = " + token.getText().replace("\n",    "\\n"));
        n++;
    }
 }
 }

上面的类(来自同一个教程)不产生任何输出.然而，如果我在令牌循环之前插入一个 tokens.toString() ，那么输出将按预期打印.

The class above ( from the same tutorial ) does NOT produce any output. If I however insert a tokens.toString() prior to the token loop then output is printed as expected.

注意:我在带有 JDK 1.7/64 位的 Windows 7 上使用 ANTLWorks 1.4.3、ANTLR 3.4

Note: I use ANTLWorks 1.4.3, ANTLR 3.4, on Windows 7 with JDK 1.7/64bit

问题:我不明白这一点.请解释.应该有一种方法可以在没有 tokens.toString() 的情况下使其正常工作

推荐答案

CommonTokenStream extends BufferedTokenStream 有一个 List调用 getTokens() 时返回的令牌.但是这个 List令牌 仅在特定时间填充.在 3.3 和 3.4 中，它不会不会发生在 getTokens() 之后，其中 3.2 会 填充 tokens 列表.

ANTLR 3.2(及之前)

CommonTokenStream extends BufferedTokenStream which has a List<Token> tokens that is returned when one calls getTokens(). But this List<Token> tokens only gets filled at certain times. In 3.3 and 3.4 it does not happen after getTokens() where 3.2 does fill the tokens list.

public List getTokens() {
    if ( p == -1 ) {
        fillBuffer();
    }
    return tokens;
}

protected void fillBuffer() {
    // fill `tokens`
}

ANTLR 3.3(及之后)

public List getTokens() { 
    return tokens; 
}

public void fill() {
    // fill `tokens`
}

请注意 3.2 的 fill 方法是如何受保护的，并且在 3.3+ 中它是公共的，因此以下有效:

Notice how 3.2's fill method is protected and in 3.3+ it is public, so the following works:

import org.antlr.runtime.*;

public class CSVLexerTest {

  public static void main(String[] args) throws Exception {

    // the input source  
    String source =
        "val1, value2, value3, value3.2" + "\n" + 
        "\"line\nbreak\",ABAbb,end";

    // create an instance of the lexer  
    CSVLexer lexer = new CSVLexer(new ANTLRStringStream(source));

    // wrap a token-stream around the lexer and fill the tokens-list 
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    tokens.fill();

    // traverse the tokens and print them to see if the correct tokens are created
    // tokens.toString();
    int n = 1;
    for (Object o : tokens.getTokens()) {
      CommonToken token = (CommonToken) o;
      System.out.println("token(" + n + ") = " + token.getText().replace("\n",    "\\n"));
      n++;
    }
  }
}

产生输出:

token(1) = val1
token(2) = ,
token(3) =  value2
token(4) = ,
token(5) =  value3
token(6) = ,
token(7) =  value3.2
token(8) = \n
token(9) = "line\nbreak"
token(10) = ,
token(11) = ABAbb
token(12) = ,
token(13) = end
token(14) = <EOF>

这篇关于ANTLR API 问题；提供示例+解决方法；需要解释的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！