在没有语法文件的情况下编写自定义Xtext/ANTLR词法分析器 [英] Writing a custom Xtext/ANTLR lexer without a grammar file

问题描述

我正在为CoffeeScript写一个Eclipse/Xtext插件，我意识到我可能需要为此手工编写一个词法分析器. CoffeeScript解析器还使用手写词法分析器来处理缩进和其他语法技巧.

I'm writing an Eclipse/Xtext plugin for CoffeeScript, and I realized I'll probably need to write a lexer for it by hand. CoffeeScript parser also uses a hand-written lexer to handle indentation and other tricks in the grammar.

Xtext生成扩展了org.eclipse.xtext.parser.antlr.Lexer的类，而该类又扩展了org.antlr.runtime.Lexer.因此，我想将其扩展.我可以看到两种方法

Xtext generates a class that extends org.eclipse.xtext.parser.antlr.Lexer which in turn extends org.antlr.runtime.Lexer. So I suppose I'll have extend it. I can see two ways to do that

• 覆盖mTokens().这是通过生成的代码更改内部状态来完成的.
• 覆盖nextToken()似乎是很自然的方法，但是我必须保持对内部状态的了解.

• Override mTokens(). This is done by the generated code, changing the internal state.
• Override nextToken() which seems a natural approach, but then I'll have to keep track of the internal state.

我找不到任何示例，即使没有语法文件也无法为ANTLR编写简单的词法分析器.因此，最简单的答案就是指向一个指针.

I couldn't find any example how to write even a simple lexer for ANTLR without a grammar file. So the easiest answer would be a pointer to one.

对> Xtext:具有重要/语义空白的语言的语法的答案指处理缩进问题的 todotext 通过更改基础输入流中的令牌.我不想走那条路，因为要处理咖啡脚本语法的其他技巧将很困难.

An answer to Xtext: grammar for language with significant/semantic whitespace refers to todotext which handles the problem of indentation by changing the tokens in the underlying input stream. I don't want to go that way, because it would be difficult to handle other tricks of the coffeescript grammar.

更新:

同时，我意识到我的问题部分是针对Xtext的.

I realized in the meantime that my question was partly Xtext specific.

推荐答案

这就是我所做的-并且可以正常工作.

Here is what I did -- and it works.

public class MyLexer extends myprj.parser.antlr.internal.InternalMylangLexer {
  private SomeExternalLexer externalLexer;

  public Lexer(CharStream in) {
    super(in);
    externalLexer = new SomeExternalLexer(in);
  }

  @Override
  public Token nextToken() {
    Token token = null;
    ExternalToken extToken = null;
    try {
      extToken = externalLexer.nextToken();
      if (extToken == null) {
        token = CommonToken.INVALID_TOKEN;
      }
      else {
        token = mapExternalToken(extToken);
      }
    }
    catch (Exception e) {
      token = CommonToken.INVALID_TOKEN;
    }
    return token;
  }

  protected Token mapExternalToken(ExternalToken extToken) {
    // ...
  }
}

然后我有一个稍微自定义的解析器，其中包含:

Then I have a slightly customized parser containing:

public class BetterParser extends MylangParser {
  @Override
  protected TokenSource createLexer(CharStream stream) {
    MyLexer lexer = new MyLexer(stream);
    return lexer;
  }
}

我还必须更改我的MylangRuntimeModule.java以包含此方法

I also had to change my MylangRuntimeModule.java to contain this method

@Override
public Class<? extends org.eclipse.xtext.parser.IParser> bindIParser() {
     return myprj.parser.BetterParser.class ;
}

就是这样.

这篇关于在没有语法文件的情况下编写自定义Xtext/ANTLR词法分析器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！