ANTLR4:如何注入令牌 [英] ANTLR4: How to inject tokens

问题描述

我正在尝试为DSL实现预处理器，该预处理器以代码/扩展名为CPP的示例为模型.但是，我没有使用令牌工厂.需要一个吗?调用emit(token)不会按预期将令牌注入令牌流中.

I'm trying to implement a preprocessor for a DSL, modeled after the CPP example in code/extras. However, I'm not using token factory. Is one required? Calling emit(token) does not inject the tokens into the tokens stream as expected.

这是词法分析器:

// string-delimited path  
SPATH     :  '"' (~[\n\r])*? '"'
                {
                 emit();  // inject the current token
                 // launch another lexer on the include file, get tokens,
                 // emit them all at once here
                 List<CommonToken> tokens = Preprocessor.include(getText());
                 if (null != tokens) {
                   for (CommonToken tok : tokens) {
                     emit(tok);
                   }
                 }
               }
      ;

这是include方法:

Here's the include method:

@SuppressWarnings("unchecked")
public static List<CommonToken> include(String filename) {
    List<CommonToken> tokens = null;
    try (FileReader fr = openFile(filename.substring(1, filename.length() - 1));
            BufferedReader br = new BufferedReader(fr)) {
        ANTLRInputStream input = new ANTLRInputStream(br);
        PreprocessorLexer lexer = new PreprocessorLexer(input);

        tokens = (List<CommonToken>) lexer.getAllTokens();

    } catch (IOException ioe) {
        log.error("Can't load ~{}~", ioe.getLocalizedMessage());
    }
    return tokens;
}

推荐答案

您需要重写Lexer.nextToken才能提供此功能.在词法分析器中，保留Deque<Token>尚未由nextToken返回的注入令牌.当队列为空时，您的nextToken实现应根据超类实现返回下一个标记.

You need to override Lexer.nextToken to provide this feature. In your lexer, keep a Deque<Token> of injected tokens that have not yet been returned by nextToken. When the queue is empty, your implementation of nextToken should return the next token according to the superclass implementation.

这是一些示例代码.我没有尝试编译或运行它，因此它可能并不完美.

Here's some sample code. I have not tried to compile or run it so it might not be perfect.

private final Deque<Token> pendingTokens = new ArrayDeque<>();

@Override
public Token nextToken() {
    Token pending = pendingTokens.pollFirst();
    if (pending != null) {
        return pending;
    }

    Token next = super.nextToken();
    pending = pendingTokens.pollFirst();
    if (pending != null) {
        pendingTokens.addLast(next);
        return pending;
    }

    return next;
}

这篇关于ANTLR4:如何注入令牌的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！