使用lxml.etree.iterparse解析损坏的XML [英] Parsing broken XML with lxml.etree.iterparse
问题描述
我正在尝试以内存高效的方式(例如从磁盘延迟流式传输而不是将整个文件加载到内存中)来使用lxml解析巨大的xml文件.不幸的是，该文件包含一些错误的ascii字符，这些字符破坏了默认解析器.如果我设置了restore = True，则解析器可以工作，但是iterparse方法不采用restore参数或自定义解析器对象.有人知道如何使用iterparse解析损坏的xml吗?
I'm trying to parse a huge xml file with lxml in a memory efficient manner (ie streaming lazily from disk instead of loading the whole file in memory). Unfortunately, the file contains some bad ascii characters that break the default parser. The parser works if I set recover=True, but the iterparse method doesn't take the recover parameter or a custom parser object. Does anyone know how to use iterparse to parse broken xml?
#this works, but loads the whole file into memory
parser = lxml.etree.XMLParser(recover=True) #recovers from bad characters.
tree = lxml.etree.parse(filename, parser)
#how do I do the equivalent with iterparse? (using iterparse so the file can be streamed lazily from disk)
context = lxml.etree.iterparse(filename, tag='RECORD')
#record contains 6 elements that I need to extract the text from
感谢您的帮助！
编辑-这是我遇到的编码错误类型的示例:
EDIT -- Here is an example of the types of encoding errors I'm running into:
In [17]: data
Out[17]: '\t<articletext><p>The cafeteria rang with excited voices. Our barbershop quartet, The Bell \r Tones was asked to perform at the local Home for the Blind in the next town. We, of course, were glad to entertain such a worthy group and immediately agreed . One wag joked, "Which uniform should we wear?" followed with, "Oh, that\'s right, they\'ll never notice." The others didn\'t respond to this, in fact, one said that we should wear the nicest outfit we had.</p><p>A small stage was set up for us and a pretty decent P.A. system was donated for the occasion. The audience was made up of blind persons of every age, from the thirties to the nineties. Some sported sighted companions or nurses who stood or sat by their side, sharing the moment equally. I observed several German shepherds lying at their feet, adoration showing in their eyes as they wondered what was going on. After a short introduction in which we identified ourselves, stating our voice part and a little about our livelihood, we began our program. Some songs were completely familiar and others, called "Oh, yeah" songs, only the chorus came to mind. We didn\'t mind at all that some sang along \x1e they enjoyed it so much.</p><p>In fact, a popular part of our program is when the audience gets to sing some of the old favorites. The harmony parts were quite evident as they tried their voices to the different parts. I think there was more group singing in the old days than there is now, but to blind people, sound and music is more important. We received a big hand at the finale and were made to promise to return the following year. Everyone was treated to coffee and cake, our quartet going around to the different circles of friends to sing a favorite song up close and personal. As we approached a new group, one blind lady amazed me by turning to me saying, "You\'re the baritone, aren\'t you?" Previously no one had ever been able to tell which singer sang which part but this lady was listening with her whole heart.</p><p>Retired portrait photographer. Main hobby - quartet singing.</p></articletext>\n'
In [18]: lxml.etree.from
lxml.etree.fromstring lxml.etree.fromstringlist
In [18]: lxml.etree.fromstring(data)
---------------------------------------------------------------------------
XMLSyntaxError Traceback (most recent call last)
/mnt/articles/<ipython console> in <module>()
/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree.fromstring (src/lxml/lxml.etree.c:48270)()
/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:71812)()
/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._parseDoc (src/lxml/lxml.etree.c:70673)()
/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:67442)()
/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:63824)()
/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:64745)()
/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)()
XMLSyntaxError: PCDATA invalid Char value 30, line 1, column 1190
In [19]: chardet.detect(data)
Out[19]: {'confidence': 1.0, 'encoding': 'ascii'}
如您所见，chardet认为它是一个ascii文件，但是在此示例中间有一个"\ x1e"，这使lxml引发异常.
As you can see, chardet thinks it is an ascii file, but there is a "\x1e" right in the middle of this example which is making lxml raise an exception.
推荐答案
我通过创建一个具有File类对象界面的类来解决了这个问题.该类的read()方法从文件中读取一行，并替换所有不良字符"，然后再将该行返回iterparse.
I solved the problem by creating a class with a File like object interface. The class' read() method reads a line from the file and replaces any "bad characters" before returning the line to iterparse.
#psudo code
class myFile(object):
def __init__(self, filename):
self.f = open(filename)
def read(self, size=None):
return self.f.next().replace('\x1e', '').replace('some other bad character...' ,'')
#iterparse
context = lxml.etree.iterparse(myFile('bigfile.xml', tag='RECORD')
我不得不编辑myFile类几次，添加了更多的replace()调用，这些调用使lxml变得更加困难.我认为lxml的SAX解析也可以正常工作(似乎支持restore选项)，但是这种解决方案却很吸引人！
I had to edit the myFile class a few times adding some more replace() calls for a few other characters that were making lxml choke. I think lxml's SAX parsing would have worked as well (seems to support the recover option), but this solution worked like a charm!
这篇关于使用lxml.etree.iterparse解析损坏的XML的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
