ANTLR:Unicode 字符扫描 [英] ANTLR: Unicode Character Scanning

问题描述

问题:无法正确打印 Unicode 字符.

Problem: Can't get Unicode character to print correctly.

这是我的语法:

options { k=1; filter=true;
 // Allow any char but \uFFFF (16 bit -1)
charVocabulary='\u0000'..'\uFFFE'; 
}

ANYCHAR :'$'
|    '_' { System.out.println("Found underscore: "+getText()); }
|    'a'..'z' { System.out.println("Found alpha: "+getText()); }
|    '\u0080'..'\ufffe' { System.out.println("Found unicode: "+getText()); }
; 

调用词法分析器的主要方法的代码片段:

Code snippet of main method invoking the lexer:

public static void main(String[] args) {
SimpleLexer simpleLexer = new SimpleLexer(System.in);
while(true) {
try {
Token t = simpleLexer.nextToken();
System.out.println("Token : "+t);

} catch(Exception e) {}

}
}

对于输入"ठ"，我得到以下输出:

For input "ठ", I'm getting the following output :

Found unicode: 
Token : ["à",<5>,line=1,col=7]
Found unicode: 
Token : ["¤",<5>,line=1,col=8]
Found unicode:  
Token : [" ",<5>,line=1,col=9]

似乎词法分析器将 Unicode 字符ठ"视为三个单独的字符.我的目标是扫描并打印ठ".

It appears that the lexer is treating Unicode char "ठ" as three separate character. My aim is to scan and print "ठ".

推荐答案

您的问题不在于 ANTLR 生成的词法分析器，而在于您传递给它的 Java 流.该流仅读取字节(不以编码方式解释它们)，您看到的是 UTF-8 序列.

Your problem is not in the ANTLR generated lexer, but in the Java stream you pass to it. The stream reads bytes only (doesn't interpret them in an encoding), and what you see is an UTF-8 sequence.

如果是 ANTLR 3，您可以使用 ANTLRInputStream 构造函数编码作为参数:

If its ANTLR 3, you can use the ANTLRInputStream constructor that takes an ancoding as a parameter:

ANTLRInputStream (InputStream input, String encoding) throws IOException

这篇关于ANTLR:Unicode 字符扫描的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！