使用XMLReader读取大于1GB的文件时出现问题 [英] Problem reading files greater than 1GB with XMLReader

问题描述

XMLReader是否可以处理最大文件大小?

Is there a maximum file size the XMLReader can handle?

我正在尝试处理大约3GB大的XML提要.由于脚本可以正常运行并在运行后成功加载到数据库，因此肯定没有PHP错误.

I'm trying to process an XML feed about 3GB large. There are certainly no PHP errors as the script runs fine and successfully loads to the database after it's been run.

该脚本在较小的测试提要(1GB及以下)中也可以正常运行.但是，当处理较大的提要时，脚本将在大约1GB后停止读取XML文件，并继续运行脚本的其余部分.

The script also runs fine with smaller test feeds - 1GB and below. However, when processing larger feeds the script stops reading the XML File after about 1GB and continues running the rest of the script.

有人遇到过类似的问题吗?如果是这样，您如何解决呢?

Has anybody experienced a similar problem? and if so how did you work around it?

谢谢.

推荐答案

我最近遇到过类似的问题，因此我想分享自己的经验.

I had same kind of problem recently and I thought to share my experience.

无论是支持64位文件大小/偏移还是仅支持32位，似乎问题都在于PHP的编译方式.

It seems that problem is in the way PHP was compiled, whether it was compiled with support for 64bit file sizes/offsets or only with 32bit.

使用32位时，您只能处理4GB的数据.您可以在此处找到一些令人困惑但很好的解释: http: //blog.mayflower.de/archives/131-Handling-large-files-without-PHP.html

With 32bits you can only address 4GB of data. You can find a bit confusing but good explanation here: http://blog.mayflower.de/archives/131-Handling-large-files-without-PHP.html

我不得不使用Perl实用程序xml_split分割文件，您可以在这里找到: http://search.cpan.org/~mirod/XML-Twig/tools/xml_split/xml_split

I had to split my files with Perl utility xml_split which you can find here: http://search.cpan.org/~mirod/XML-Twig/tools/xml_split/xml_split

我用它来将巨大的XML文件拆分为可管理的块.该工具的优点是它将XML文件拆分为整个元素.不幸的是，它不是很快.

I used it to split my huge XML file into manageable chunks. The good thing about the tool is that it splits XML files over whole elements. Unfortunately its not very fast.

我只需要这样做一次就可以满足我的需要，但是我不建议您重复使用它.拆分后，我在大小小于1GB的较小文件上使用了XMLReader.

I needed to do this one time only and it suited my needs, but I wouldn't recommend it repetitive use. After splitting I used XMLReader on smaller files of about 1GB in size.

这篇关于使用XMLReader读取大于1GB的文件时出现问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！