在Python中加载大型JSON列表的最佳方法是什么? [英] What's the best way to load large JSON lists in Python?

问题描述

我可以访问一组文件(每个文件大约80-800mb).不幸的是，每个文件中只有一行.该行仅包含一个JSON对象(列表列表).将其加载并解析为较小的JSON对象的最佳方法是什么?

I have access to a set of files (around 80-800mb each). Unfortunately, there's only one line in every file. The line contains exactly one JSON object (a list of lists). What's the best way to load and parse it into smaller JSON objects?

推荐答案

此处已有类似的帖子.这是他们提出的解决方案:

There is already a similar post here. Here is the solution they proposed:

import json
with open('file.json') as infile:
  o = json.load(infile)
  chunkSize = 1000
  for i in xrange(0, len(o), chunkSize):
    with open('file_' + str(i//chunkSize) + '.json', 'w') as outfile:
      json.dump(o[i:i+chunkSize], outfile)

这篇关于在Python中加载大型JSON列表的最佳方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！