ANTLR4:如何注入代币 [英] ANTLR4: How to inject tokens

问题描述

我正在尝试为 DSL 实现一个预处理器，以代码/附加中的 CPP 示例为模型.但是，我没有使用令牌工厂.需要一个吗?调用emit(token) 不会按预期将令牌注入令牌流.

I'm trying to implement a preprocessor for a DSL, modeled after the CPP example in code/extras. However, I'm not using token factory. Is one required? Calling emit(token) does not inject the tokens into the tokens stream as expected.

这是词法分析器:

// string-delimited path  
SPATH     :  '"' (~[\n\r])*? '"'
                {
                 emit();  // inject the current token
                 // launch another lexer on the include file, get tokens,
                 // emit them all at once here
                 List<CommonToken> tokens = Preprocessor.include(getText());
                 if (null != tokens) {
                   for (CommonToken tok : tokens) {
                     emit(tok);
                   }
                 }
               }
      ;

这是包含方法:

@SuppressWarnings("unchecked")
public static List<CommonToken> include(String filename) {
    List<CommonToken> tokens = null;
    try (FileReader fr = openFile(filename.substring(1, filename.length() - 1));
            BufferedReader br = new BufferedReader(fr)) {
        ANTLRInputStream input = new ANTLRInputStream(br);
        PreprocessorLexer lexer = new PreprocessorLexer(input);

        tokens = (List<CommonToken>) lexer.getAllTokens();

    } catch (IOException ioe) {
        log.error("Can't load ~{}~", ioe.getLocalizedMessage());
    }
    return tokens;
}

推荐答案

您需要覆盖 Lexer.nextToken 才能提供此功能.在您的词法分析器中，保留 nextToken 尚未返回的注入令牌的 Deque.当队列为空时，您的 nextToken 实现应根据超类实现返回下一个标记.

You need to override Lexer.nextToken to provide this feature. In your lexer, keep a Deque<Token> of injected tokens that have not yet been returned by nextToken. When the queue is empty, your implementation of nextToken should return the next token according to the superclass implementation.

这是一些示例代码.我没有尝试编译或运行它，所以它可能不完美.

Here's some sample code. I have not tried to compile or run it so it might not be perfect.

private final Deque<Token> pendingTokens = new ArrayDeque<>();

@Override
public Token nextToken() {
    Token pending = pendingTokens.pollFirst();
    if (pending != null) {
        return pending;
    }

    Token next = super.nextToken();
    pending = pendingTokens.pollFirst();
    if (pending != null) {
        pendingTokens.addLast(next);
        return pending;
    }

    return next;
}

这篇关于ANTLR4:如何注入代币的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！