ANTLR：Unicode字符扫描 [英] ANTLR: Unicode Character Scanning

问题描述

问题：无法正确打印Unicode字符。

Problem: Can't get Unicode character to print correctly.

这是我的语法：

options { k=1; filter=true;
 // Allow any char but \uFFFF (16 bit -1)
charVocabulary='\u0000'..'\uFFFE'; 
}

ANYCHAR :'$'
|    '_' { System.out.println("Found underscore: "+getText()); }
|    'a'..'z' { System.out.println("Found alpha: "+getText()); }
|    '\u0080'..'\ufffe' { System.out.println("Found unicode: "+getText()); }
; 

调用词法分析器的主要方法的代码片段：

Code snippet of main method invoking the lexer:

public static void main(String[] args) {
SimpleLexer simpleLexer = new SimpleLexer(System.in);
while(true) {
try {
Token t = simpleLexer.nextToken();
System.out.println("Token : "+t);

} catch(Exception e) {}

}
}

对于输入ठ，我得到以下输出：

For input "ठ", I'm getting the following output :

Found unicode: 
Token : ["à",<5>,line=1,col=7]
Found unicode: 
Token : ["¤",<5>,line=1,col=8]
Found unicode:  
Token : [" ",<5>,line=1,col=9]

似乎词法分析器将Unicode字符ठ视为三个单独的字符。我的目标是扫描并打印ठ。

It appears that the lexer is treating Unicode char "ठ" as three separate character. My aim is to scan and print "ठ".

推荐答案

您的问题不在ANTLR生成的词法分析器中，而是在Java中流传递给它。流只读取字节（不会在编码中解释它们），你看到的是UTF-8序列。

Your problem is not in the ANTLR generated lexer, but in the Java stream you pass to it. The stream reads bytes only (doesn't interpret them in an encoding), and what you see is an UTF-8 sequence.

如果它的ANTLR 3，你可以使用 ANTLRInputStream 构造函数，它将ancoding作为参数：

If its ANTLR 3, you can use the ANTLRInputStream constructor that takes an ancoding as a parameter:

ANTLRInputStream (InputStream input, String encoding) throws IOException

这篇关于ANTLR：Unicode字符扫描的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！