有没有一种内存高效且快速的方法来在python中加载大json文件? [英] Is there a memory efficient and fast way to load big json files in python?

问题描述

我有一些500MB的json文件. 如果我使用平凡"的json.load一次加载所有内容，它将消耗大量内存.

I have some json files with 500MB. If I use the "trivial" json.load to load its content all at once, it will consume a lot of memory.

是否可以部分读取文件?如果它是文本行分隔文件，则可以在行上进行迭代.我正在寻找一个比喻.

Is there a way to read partially the file? If it was a text, line delimited file, I would be able to iterate over the lines. I am looking for analogy to it.

有什么建议吗? 谢谢

推荐答案

简短答案:否.

正确分割json文件将使您对json对象图有深入的了解.

Properly dividing a json file would take intimate knowledge of the json object graph to get right.

但是，如果您掌握了这些知识，则可以实现一个类似文件的对象，该对象将json文件包装起来并吐出适当的块.

However, if you have this knowledge, then you could implement a file-like object that wraps the json file and spits out proper chunks.

例如，如果您知道json文件是单个对象数组，则可以创建一个生成器，该生成器包装json文件并返回该数组的大块.

For instance, if you know that your json file is a single array of objects, you could create a generator that wraps the json file and returns chunks of the array.

您必须进行一些字符串内容解析才能正确获取json文件的分块.

You would have to do some string content parsing to get the chunking of the json file right.

我不知道是什么生成了您的json内容.如果可能的话，我会考虑生成许多可管理的文件，而不是一个大文件.

I don't know what generates your json content. If possible, I would consider generating a number of managable files, instead of one huge file.

这篇关于有没有一种内存高效且快速的方法来在python中加载大json文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！