迭代解析JSON文件 [英] Iteratively parse JSON file

问题描述

我有1000多个类似的JSON文件

I have 1000+ JSON files that look like

{
    "name": "Some name",
    "part_num": "123456",

    "other_config": {
        // Large amount of objects
    },
    "some more": {
        // Large amount of objects
    }
    // etc
}

当我的程序启动时，它必须扫描所有这些JSON文件的目录，并加载每个文件，然后提取"name"和"part_num"值，并使用这些值填充列表视图.然后，用户选择一个，然后重新解析该配置，然后采取适当的操作.

When my program starts up, it has to scan the directory with all these JSON files, load each one, and extract the "name" and "part_num" values and populates a list view with these values. The user then selects one and that config is then re-parsed and the proper actions taken then.

问题在于读取许多文件需要一段时间.我已经通过使用multiprocessing在后台将所有可用内核上投入工作，然后在完成后填充列表视图的方式上有所缓解，但是我仍然受IO的束缚.由于我知道此代码将在处理器和硬盘驱动器比我的慢的计算机上运行，​​因此这种速度是不可接受的.

The problem is reading that many files takes a while. I've mitigated it somewhat by using multiprocessing to throw the work on all available cores in the background, then populating the list view once done, but I'm still bounded by IO. Since I know this code will be running on computers with slower processors and hard drives than mine, this speed is not acceptable.

一般情况是，我需要的值位于文件的开头，但我不能认为在最坏的情况下会出现这种情况. 是否可以迭代解析JSON文件，以便我可以更快地从这些文件中加载所需的内容?

The average case scenario is that the values I need are at the beginning of the file, but I can't assume that in the worst case. Is there a way to iteratively parse a JSON file so that I can load what I need from these files faster?

我可以求助于正则表达式，但我真的不愿意这样做.

I could resort to regex, but I'd really prefer not to.

推荐答案

有执行的 ijson 以pythonc方式进行迭代json解析.

There is ijson which performs performs iterative json parsing in a pythonc way.

我想您的问题的解决方案是:

I guess that solution to your problem would be:

import ijson

f = open('...')
objects = ijson.items(f, 'other_config.item')
for part in objects:
    do_something_with(part)

免责声明，我没有使用该库.

Dislaimer I did not use that library.

这篇关于迭代解析JSON文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！