如何从大型 yaml 文件中获取部分数据? [英] How to fetch partial data from a large yaml file?

问题描述

我有一个包含一些无用数据的大型 yaml 文件.使用 yaml.load() 加载此文件时，内存消耗超出了我们计算机的物理限制.我读不懂.我是否只能读取python dict所需的部分数据?是否有一些库或代码可以解决这个问题?

I have a large yaml file containing some useless data. When using yaml.load() to load this file, memory consumption exceeds the physical limit of our computer. I can't read it. Whether I can only read partial data which I need to a python dict? are there some library or code to solve this problem?

推荐答案

使用 PyYaml，您可以执行以下操作:

Using PyYaml, you can do something like this:

with open("file.yaml", 'r') as handle:
  for event in yaml.parse(handle):
    # handle the event here

这会逐个事件处理 YAML 文件，而不是将其全部加载到数据结构中.当然，您现在需要从事件流中手动解析结构，但这允许您不进一步处理部分数据.

This processes the YAML file event by event, instead of loading it all into a data structure. Of course, you now need to parse the structure manually from the event stream, but this allows you to not process parts of the data further.

你可以看看 PyYaml 的 Composer 实现 查看它如何从事件构造 Python 对象，以及它期望从事件流中得到什么结构.

You can look at PyYaml's Composer implementation to see how it constructs Python objects from events, and what structure it expects from the event stream.

这篇关于如何从大型 yaml 文件中获取部分数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！