在 Python 中生成非常大的 XML 文件? [英] Generating very large XML files in Python?

问题描述

有谁知道在 Python 中生成非常大的 xml 文件(例如 100-500 MiB)的内存有效方法吗?

Does anyone know of a memory efficient way to generate very large xml files (e.g. 100-500 MiB) in Python?

我一直在使用 lxml，但内存使用量已经达到顶峰.

I've been utilizing lxml, but memory usage is through the roof.

推荐答案

也许您可以使用模板引擎而不是自己生成/构建 xml?

Perhaps you could use a templating engine instead of generating/building the xml yourself?

Genshi 例如基于 xml 并支持流输出.一个非常基本的例子:

Genshi for example is xml-based and supports streaming output. A very basic example:

from genshi.template import MarkupTemplate

tpl_xml = '''
<doc xmlns:py="http://genshi.edgewall.org/">
<p py:for="i in data">${i}</p>
</doc>
'''

tpl = MarkupTemplate(tpl_xml)
stream = tpl.generate(data=xrange(10000000))

with open('output.xml', 'w') as f:
    stream.render(out=f)

可能需要一段时间，但内存使用率仍然很低.

It might take a while, but memory usage remains low.

Mako 模板引擎(不是原生"xml)的相同示例，但速度要快得多:

The same example for the Mako templating engine (not "natively" xml), but a lot faster:

from mako.template import Template
from mako.runtime import Context

tpl_xml = '''
<doc>
% for i in data:
<p>${i}</p>
% endfor
</doc>
'''

tpl = Template(tpl_xml)

with open('output.xml', 'w') as f:
    ctx = Context(f, data=xrange(10000000))
    tpl.render_context(ctx)

最后一个例子在我的笔记本电脑上运行了大约 20 秒，生成了一个(公认非常简单)151 MB 的 xml 文件，完全没有内存问题.(根据 Windows 任务管理器，它保持恒定在 10MB 左右)

The last example ran on my laptop for about 20 seconds, producing a (admittedly very simple) 151 MB xml file, no memory problems at all. (according to Windows task manager it remained constant at about 10MB)

根据您的需要，这可能是一种比使用 SAX 等更友好、更快速的生成 xml 的方式......查看文档以了解您可以使用这些引擎做什么(还有其他引擎，我刚刚挑选了以这两个为例)

Depending on your needs, this might be a friendlier and faster way of generating xml than using SAX etc... Check out the docs to see what you can do with these engines (there are others as well, I just picked out these two as examples)

这篇关于在 Python 中生成非常大的 XML 文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！