使用多重处理解析非常大的XML文件 [英] Parsing Very Large XML Files Using Multiprocessing

问题描述

我有一个巨大的XML文件，我对如何处理它有点不知所措.这是60 GB，我需要阅读.

I have a huge XML file, and I'm a tad bit at a loss on how to handle it. It's 60 GBs, and I need to read it.

我在想是否可以使用多处理模块读取python文件?

I was thinking if there a way to use multiprocessing module to read the python file?

有人可以向我指出这样做的示例吗?

Does anyone have any samples of doing this that they could point me to?

谢谢

推荐答案

对于这种大小的文件，建议您使用流式XML解析器.在Python中，这将是cElementTree或lxml.etree中的iterparse方法:

For a file of that size, I suggest you use a streaming XML parser. In Python, this would be the iterparse method from cElementTree or lxml.etree:

http://effbot.org/zone/element-iterparse.htm

这篇关于使用多重处理解析非常大的XML文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！