数据集不适合内存 [英] Dataset does not fit in memory

问题描述

我有一个 MNIST 之类的数据集，它不适合内存(进程内存，非 GPU 内存).我的数据集是 4GB.

I have an MNIST like dataset that does not fit in memory, (process memory, not gpu memory). My dataset is 4GB.

这不是 TFLearn 问题.

据我所知，model.fit 需要 x 和 y 的 array.

As far as I know model.fit requires an array for x and y.

TFLearn 示例:

TFLearn example:

model.fit(x, y, n_epoch=10, validation_set=(val_x, val_y))

我想知道有没有一种方法可以传递批处理迭代器"而不是数组.基本上，对于每个批次，我都会从磁盘加载必要的数据.

I was wondering is there's a way where we can pass a "batch iterator", instead of an array. Basically for each batch I would load the necessary data from disk.

这样我就不会遇到进程内存溢出错误.

This way I would not run into process memory overflow errors.

编辑np.memmap 可能是一个选项.但我不知道如何跳过组成标题的前几个字节.

EDIT np.memmap could be an option. But I don't see how to skip the first few bytes that compose the header.

推荐答案

您可以使用 数据集 API.

数据集 API 支持多种文件格式，以便您可以处理无法放入内存的大型数据集"

"The Dataset API supports a variety of file formats so that you can process large datasets that do not fit in memory"

基本上，输入管道将成为您图表的一部分.

Basically the input pipeline would become part of your graph.

如果内存仍然是一个问题，那么您可以使用生成器来创建您的 tf.data.Dataset.此外，您可以通过准备 tfrecords 来创建数据集，从而加快流程.

If memory is still an issue then you can use a generator to create your tf.data.Dataset. Further, you could potentially make the process quicker by preparing tfrecords to create your Dataset.

这篇关于数据集不适合内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！