读取相当大的 JSON 文件 [英] Reading rather large JSON files

问题描述

可能的重复:
是否有内存高效且加载大型 JSON 文件的快速方法?

所以我有一些相当大的 json 编码文件.最小的是 300MB，但这是迄今为止最小的.其余的是多个 GB，从大约 2GB 到 10GB+ 不等.

So I have some rather large json encoded files. The smallest is 300MB, but this is by far the smallest. The rest are multiple GB, anywhere from around 2GB to 10GB+.

所以我在尝试用 Python 加载文件时似乎内存不足.我目前只是在运行一些测试来大致了解处理这些东西需要多长时间才能看到从这里开始的方向.这是我用来测试的代码:

So I seem to run out of memory when trying to load the file with Python. I'm currently just running some tests to see roughly how long dealing with this stuff is going to take to see where to go from here. Here is the code I'm using to test:

from datetime import datetime
import json

print datetime.now()

f = open('file.json', 'r')
json.load(f)
f.close()

print datetime.now()

毫不奇怪，Python 给了我一个 MemoryError.看起来 json.load() 调用了 json.loads(f.read())，它试图首先将整个文件转储到内存中，这显然是行不通的.

Not too surprisingly, Python gives me a MemoryError. It appears that json.load() calls json.loads(f.read()), which is trying to dump the entire file into memory first, which clearly isn't going to work.

有什么办法可以干净利落地解决这个问题?

Any way I can solve this cleanly?

我知道这是旧的，但我不认为这是重复的.虽然答案是一样的，但问题是不同的.在重复"中，问题是如何有效地读取大文件，而这个问题涉及根本不适合内存的文件.不需要效率.

I know this is old, but I don't think this is a duplicate. While the answer is the same, the question is different. In the "duplicate", the question is how to read large files efficiently, whereas this question deals with files that won't even fit in to memory at all. Efficiency isn't required.

推荐答案

这里的问题是，JSON 作为一种格式，一般都是全解析然后在内存中处理，对于这么大的数据量来说显然是有问题.

The issue here is that JSON, as a format, is generally parsed in full and then handled in-memory, which for such a large amount of data is clearly problematic.

解决方案是将数据作为流处理 - 读取文件的一部分，使用它，然后重复.

The solution to this is to work with the data as a stream - reading part of the file, working with it, and then repeating.

最好的选择似乎是使用类似 ijson 之类的东西——一个可以使用的模块JSON 作为流，而不是块文件.

The best option appears to be using something like ijson - a module that will work with JSON as a stream, rather than as a block file.

也值得一看 - kashif 的评论 关于 json-streamer 和 Henrik Heino 的评论关于bigjson.

Also worth a look - kashif's comment about json-streamer and Henrik Heino's comment about bigjson.

这篇关于读取相当大的 JSON 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！