编写没有语法文件的自定义 Xtext/ANTLR 词法分析器 [英] Writing a custom Xtext/ANTLR lexer without a grammar file
问题描述
我正在为 CoffeeScript 编写一个 Eclipse/Xtext 插件,我意识到我可能需要手动为它编写一个词法分析器.CoffeeScript 解析器还使用 手写词法分析器处理语法中的缩进和其他技巧.
I'm writing an Eclipse/Xtext plugin for CoffeeScript, and I realized I'll probably need to write a lexer for it by hand. CoffeeScript parser also uses a hand-written lexer to handle indentation and other tricks in the grammar.
Xtext 生成一个扩展 org.eclipse.xtext.parser.antlr.Lexer
的类,而后者又扩展了 org.antlr.runtime.Lexer
.所以我想我会延长它.我可以看到两种方法来做到这一点
Xtext generates a class that extends org.eclipse.xtext.parser.antlr.Lexer
which in turn extends org.antlr.runtime.Lexer
. So I suppose I'll have extend it. I can see two ways to do that
- 覆盖
mTokens()
.这是由生成的代码完成的,更改内部状态. - 覆盖
nextToken()
这似乎是一种自然的方法,但我必须跟踪内部状态.
- Override
mTokens()
. This is done by the generated code, changing the internal state. - Override
nextToken()
which seems a natural approach, but then I'll have to keep track of the internal state.
我找不到任何示例,如何在没有语法文件的情况下为 ANTLR 编写一个简单的词法分析器.所以最简单的答案是指向一个.
I couldn't find any example how to write even a simple lexer for ANTLR without a grammar file. So the easiest answer would be a pointer to one.
Xtext 的答案:具有重要/语义空白的语言的语法 指的是处理缩进问题的 todotext通过更改底层输入流中的标记.我不想走那条路,因为处理咖啡脚本语法的其他技巧会很困难.
An answer to Xtext: grammar for language with significant/semantic whitespace refers to todotext which handles the problem of indentation by changing the tokens in the underlying input stream. I don't want to go that way, because it would be difficult to handle other tricks of the coffeescript grammar.
更新:
同时我意识到我的问题部分是针对 Xtext 的.
I realized in the meantime that my question was partly Xtext specific.
推荐答案
这是我所做的 - 并且有效.
Here is what I did -- and it works.
public class MyLexer extends myprj.parser.antlr.internal.InternalMylangLexer {
private SomeExternalLexer externalLexer;
public Lexer(CharStream in) {
super(in);
externalLexer = new SomeExternalLexer(in);
}
@Override
public Token nextToken() {
Token token = null;
ExternalToken extToken = null;
try {
extToken = externalLexer.nextToken();
if (extToken == null) {
token = CommonToken.INVALID_TOKEN;
}
else {
token = mapExternalToken(extToken);
}
}
catch (Exception e) {
token = CommonToken.INVALID_TOKEN;
}
return token;
}
protected Token mapExternalToken(ExternalToken extToken) {
// ...
}
}
然后我有一个稍微定制的解析器,其中包含:
Then I have a slightly customized parser containing:
public class BetterParser extends MylangParser {
@Override
protected TokenSource createLexer(CharStream stream) {
MyLexer lexer = new MyLexer(stream);
return lexer;
}
}
我还必须更改我的 MylangRuntimeModule.java
以包含此方法
I also had to change my MylangRuntimeModule.java
to contain this method
@Override
public Class<? extends org.eclipse.xtext.parser.IParser> bindIParser() {
return myprj.parser.BetterParser.class ;
}
就是这样.
这篇关于编写没有语法文件的自定义 Xtext/ANTLR 词法分析器的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!