JSON 文件中的元音变音会导致 ANTLR4 创建的 Python 代码出错 [英] Umlauts in JSON files lead to errors in Python code created by ANTLR4

问题描述

我已经使用 github/antlr4 上的 JSON 语法创建了 python 模块

I've created python modules from the JSON grammar on github / antlr4 with

antlr4 -Dlanguage=Python3 JSON.g4

我按照本指南编写了一个主程序JSON2.py":https://github.com/antlr/antlr4/blob/master/doc/python-target.md并从github下载了example1.json.

I've written a main program "JSON2.py" following this guide: https://github.com/antlr/antlr4/blob/master/doc/python-target.md and downloaded the example1.json also from github.

python3 ./JSON2.py example1.json # works perfectly, but 
python3 ./JSON2.py bookmarks-2017-05-24.json # the bookmarks contain German Umlauts like "ü"

...
File "/home/xyz/lib/python3.5/site-packages/antlr4/FileStream.py", line 27, in readDataFrom
    return codecs.decode(bytes, encoding, errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 227: ordinal not in range(128)

JSON2.py 中的违规行是

The offending line in JSON2.py is

input = FileStream(argv[1])

我搜索了 stackoverflow 并尝试了这个而不是使用上面的 FileStream:

I've searched stackoverflow and tried this instead of using the above FileStream:

fp = codecs.open(argv[1], 'rb', 'utf-8')
try:
    input = fp.read()
finally:
    fp.close()
lexer = JSONLexer(input)
stream = CommonTokenStream(lexer)
parser = JSONParser(stream)
tree = parser.json() # This is line 39, mentioned in the error message     

即使输入文件不包含变音符号，此程序的执行也会以错误消息结束:

Execution of this program ends with an error message, even if the input file doesn't contain Umlauts:

python3 ./JSON2.py example1.json 
Traceback (most recent call last):
  File "./JSON2.py", line 46, in <module>
    main(sys.argv)
  File "./JSON2.py", line 39, in main
    tree = parser.json()    
  File "/home/x/Entwicklung/antlr/links/JSONParser.py", line 108, in json
    self.enterRule(localctx, 0, self.RULE_json)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/Parser.py", line 358, in enterRule
    self._ctx.start = self._input.LT(1)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/CommonTokenStream.py", line 61, in LT
    self.lazyInit()
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 186, in lazyInit
    self.setup()
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 189, in setup
    self.sync(0)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 111, in sync
    fetched = self.fetch(n)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 123, in fetch
    t = self.tokenSource.nextToken()
  File "/home/xyz/lib/python3.5/site-packages/antlr4/Lexer.py", line 111, in nextToken
    tokenStartMarker = self._input.mark()
AttributeError: 'str' object has no attribute 'mark'

这解析正确:

<代码>javac *.javagrun JSON json -gui bookmarks-2017-05-24.json所以语法本身不是问题.

javac *.java grun JSON json -gui bookmarks-2017-05-24.json So the grammar itself is not the problem.

最后的问题是:我应该如何在python中处理输入文件，以便词法分析器和解析器能够消化它?

So finally the question: How should I process the input file in python, so that lexer and parser can digest it?

提前致谢.

推荐答案

我通过传递encoding信息解决了这个问题:

I solved it by passing the encoding info:

input = FileStream(sys.argv[1], encoding = 'utf8')

如果没有编码信息，我会遇到和你一样的问题.

If without the encoding info, I will have the same issue as yours.

Traceback (most recent call last):
  File "test.py", line 20, in <module>
    main()
  File "test.py", line 9, in main
    input = FileStream(sys.argv[1])
  File ".../lib/python3.5/site-packages/antlr4/FileStream.py", line 20, in __init__
    super().__init__(self.readDataFrom(fileName, encoding, errors))
  File ".../lib/python3.5/site-packages/antlr4/FileStream.py", line 27, in readDataFrom
    return codecs.decode(bytes, encoding, errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

我的输入数据在哪里<代码>[今明]天(台南|高雄)的?天气如何

这篇关于JSON 文件中的元音变音会导致 ANTLR4 创建的 Python 代码出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！