在侦听器中使用 ParserRuleContext 遍历令牌 - ANTLR4 [英] Traversal of tokens using ParserRuleContext in listener - ANTLR4

查看:22
本文介绍了在侦听器中使用 ParserRuleContext 遍历令牌 - ANTLR4的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

在使用侦听器迭代令牌时,我想知道如何使用 ParserRuleContext 来查看令牌流中的下一个令牌或接下来的几个令牌?

While iterating over the tokens using a Listener, I would like to know how to use the ParserRuleContext to peek at the next token or the next few tokens in the token stream?

在下面的代码中,我试图查看当前令牌之后直到 EOF 的所有令牌:

In the code below I am trying to peek at all the tokens after the current token till the EOF:

@Override 
public void enterSemicolon(JavaParser.SemicolonContext ctx) {

    Token tok, semiColon = ctx.getStart();  
    int currentIndex = semiColon.getStartIndex();
    int reqInd = currentIndex+1;
    TokenSource tokSrc= semiColon.getTokenSource();
    CharStream srcStream = semiColon.getInputStream();
    srcStream.seek(currentIndex);

    while(true){

        tok = tokSrc.nextToken() ;
        System.out.println(tok);
        if(tok.getText()=="<EOF>"){break;}
        srcStream.seek(reqInd++);
    }
}

但我得到的输出是:

            .
            .
            .
            .
            .
[@-1,131:130='',<-1>,13:0]
[@-1,132:131='',<-1>,13:0]
[@-1,133:132='',<-1>,13:0]
[@-1,134:133='',<-1>,13:0]
[@-1,135:134='',<-1>,13:0]
[@-1,136:135='',<-1>,13:0]
[@-1,137:136='',<-1>,13:0]
[@-1,138:137='',<-1>,13:0]
[@-1,139:138='',<-1>,13:0]
[@-1,140:139='',<-1>,13:0]
[@-1,141:140='',<-1>,13:0]
[@-1,142:141='',<-1>,13:0]
[@-1,143:142='',<-1>,13:0]
[@-1,144:143='',<-1>,13:0]
[@-1,145:144='',<-1>,13:0]
[@-1,146:145='',<-1>,13:0]
[@-1,147:146='',<-1>,13:0]
[@-1,148:147='',<-1>,13:0]
[@-1,149:148='',<-1>,13:0]
[@-1,150:149='',<-1>,13:0]
[@-1,151:150='',<-1>,13:0]
[@-1,152:151='',<-1>,13:0]
[@-1,153:152='',<-1>,13:0]
[@-1,154:153='',<-1>,13:0]
[@-1,155:154='',<-1>,13:0]
[@-1,156:155='',<-1>,13:0]
[@-1,157:156='',<-1>,13:0]
[@-1,158:157='',<-1>,13:0]
[@-1,159:158='',<-1>,13:0]
[@-1,160:159='',<-1>,13:0]
[@-1,161:160='<EOF>',<-1>,13:0]
[@-1,137:136='',<-1>,13:0]
[@-1,138:137='',<-1>,13:0]
[@-1,139:138='',<-1>,13:0]
[@-1,140:139='',<-1>,13:0]
[@-1,141:140='',<-1>,13:0]
[@-1,142:141='',<-1>,13:0]
[@-1,143:142='',<-1>,13:0]
[@-1,144:143='',<-1>,13:0]
[@-1,145:144='',<-1>,13:0]
[@-1,146:145='',<-1>,13:0]
[@-1,147:146='',<-1>,13:0]
[@-1,148:147='',<-1>,13:0]
[@-1,149:148='',<-1>,13:0]
[@-1,150:149='',<-1>,13:0]
[@-1,151:150='',<-1>,13:0]
[@-1,152:151='',<-1>,13:0]
[@-1,153:152='',<-1>,13:0]
[@-1,154:153='',<-1>,13:0]
[@-1,155:154='',<-1>,13:0]
[@-1,156:155='',<-1>,13:0]
[@-1,157:156='',<-1>,13:0]
[@-1,158:157='',<-1>,13:0]
[@-1,159:158='',<-1>,13:0]
[@-1,160:159='',<-1>,13:0]
[@-1,161:160='<EOF>',<-1>,13:0]
            .
            .
            .
            .

我们看到,虽然我能够遍历所有令牌直到 EOF,但我无法获得令牌的实际内容或类型.我想知道是否有使用侦听器遍历的巧妙方法.

We see that although I am able to traverse through all the tokens till EOF, I unable to get the actual content or type of the tokens. I would like to know if there is a neat way of doing this using listener traversing.

推荐答案

很难确定,但

tok = tokSrc.nextToken() ;

似乎正在重新运行词法分析器,从假定的正确标记边界开始,但没有重置词法分析器.词法分析器抛出错误可能解释了观察到的行为.

appears to be rerunning the lexer, starting at a presumed proper token boundary, but without having reset the lexer. The lexer throwing errors might explain the observed behavior.

不过,更好的方法是简单地恢复现有的令牌流:

Still, a better approach would be to simply recover the existing Token stream:

public class Walker implements YourJavaListener {

    CommonTokenStream tokens;

    public Walker(JavaParser parser) {
        tokens = (CommonTokenStream) parser.getTokenStream()
    }

然后访问流以获取所需的令牌:

then access the stream to get the desired tokens:

@Override 
public void enterSemicolon(JavaParser.SemicolonContext ctx) {
    TerminalNode semi = ctx.semicolon(); // adjust as needed for your impl.
    Token tok = semi.getSymbol();
    int idx = tok.getTokenIndex();

    while(tok.getType() != IntStream.EOF) {
        System.out.println(tok);
        tok = tokens.get(idx++);
    }
}

一种完全不同的方法可能会满足您的最终目的,即直接从父上下文中获取一组有限的令牌:

An entirely different approach that might serve your ultimate purpose is to get a limited set of tokens directly from the parent context:

ParserRuleContext pctx = ctx.getParent();
List<TerminalNode> nodes = pctx.getTokens(pctx.getStart(), pctx.getStop());

这篇关于在侦听器中使用 ParserRuleContext 遍历令牌 - ANTLR4的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆