在Python中打开大型JSON文件 [英] Opening A large JSON file in Python

问题描述

当我尝试使用json.load()打开时，我有一个1.7 GB的JSON文件，那么它给出了内存错误，怎么能在python中读取json文件?

I have a 1.7 GB JSON file when I am trying to open with json.load() then it is giving memory error, How could read the json file in python?

我的JSON文件是包含特定键的大量对象.

My JSON file is a big array of objects containing specific keys.

好吧，如果它只是对象的一大数组，并且事先知道对象的结构，那么就不需要使用我们可以逐行读取它的工具.一行仅包含数组的一个元素.我注意到这是json文件的存储方式，对我来说它就像

Well if it is just one big array of objects and it is known the structure of objects beforehand then there is no need to use tools we could read it line by line. A line will just contain one element of the array. I noticed that is the way json files are stored, for me it worked as just

>>>for line in open('file.json','r').readline():
...    do something with(line) 

推荐答案

您想要一个增量json解析器，例如 yajl 和一个它的python绑定.增量解析器从输入中读取尽可能少的内容，并在解码有意义的内容时调用回调.例如，仅从大json文件中提取数字:

You want an incremental json parser like yajl and one of its python bindings. An incremental parser reads as little as possible from the input and invokes a callback when something meaningful is decoded. For example, to pull only numbers from a big json file:

class ContentHandler(YajlContentHandler):
    def yajl_number(self, ctx, val):
         list_of_numbers.append(float(val))

parser = YajlParser(ContentHandler())
parser.parse(some_file)

有关更多信息，请参见 http://pykler.github.com/yajl-py/.

See http://pykler.github.com/yajl-py/ for more info.

这篇关于在Python中打开大型JSON文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！