解析大型JSON文件 [英] Parse large JSON file

问题描述

我正在研究可访问API的cron脚本，可接收JSON文件(大量对象)并将其存储在本地.完成此操作后，另一个脚本需要解析下载的JSON文件，并将每个对象插入MySQL数据库.

I'm working on a cron script that hits an API, receives JSON file (a large array of objects) and stores it locally. Once that is complete another script needs to parse the downloaded JSON file and insert each object into a MySQL database.

我目前正在使用file_get_contents()和json_decode().这将尝试在尝试处理之前将整个文件读入内存.除了我的JSON文件通常在250MB-1GB +之间的事实之外，这将很好.我知道我可以增加我的PHP内存限制，但这在我看来似乎并不是最大的答案.我知道我可以运行fopen()和fgets()逐行读取文件，但是我需要按每个json对象读取文件.

I'm currently using a file_get_contents() along with json_decode(). This will attempt to read the whole file into memory before trying to process it. This would be fine except for the fact that my JSON files will usually range from 250MB-1GB+. I know I can increase my PHP memory limit but that doesn't seem to be the greatest answer in my mind. I'm aware that I can run fopen() and fgets() to read the file in line by line, but I need to read the file in by each json object.

有没有一种方法可以读取每个对象的文件，或者还有另一种类似的方法?

Is there a way to read in the file per object, or is there another similar approach?

推荐答案

这实际上取决于json文件包含的内容.

This really depends on what the json files contain.

如果打开文件到内存中不是一个选项，那么您唯一想到的另一个选择就是fopen/fgets.

If opening the file one shot into memory is not an option, your only other option, as you eluded to, is fopen/fgets.

可以逐行读取，并且如果这些json对象具有一致的结构，则可以轻松检测文件中的json对象何时开始和结束.

Reading line by line is possible, and if these json objects have a consistent structure, you can easily detect when a json object in a file starts, and ends.

一旦收集了整个对象，就将其插入数据库，然后继续下一个.

Once you collect a whole object, you insert it into a db, then go on to the next one.

没有更多的东西了.取决于您的数据源，用于检测json对象的开始和结束的算法可能会变得复杂，但是我之前使用复杂得多的结构(xml)进行了类似的操作，并且效果很好.

There isn't much more to it. the algorithm to detect the beginning and end of a json object may get complicating depending on your data source, but I hvae done something like this before with a far more complex structure (xml) and it worked fine.

这篇关于解析大型JSON文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！