Keras-用于图像和蒙版的大型数据集的生成器 [英] Keras - Generator for large dataset of Images and Masks

问题描述

我正在尝试构建一个同时具有输入和输出(遮罩)图像的模型. 由于数据集的大小和我有限的内存，我尝试使用Keras文档中引入的生成器方法:

I'm trying to build a Model that has Images for both its Inputs and Outputs (masks). Because of the size of the Dataset and my limited Memory, I tried using the Generator Approach introduced in the Keras Documentation:

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

model.fit_generator(
    train_generator,
    samples_per_epoch=2000,
    nb_epoch=50)

除代码到达以下行外，一切似乎都可以正常工作:

Everything seems to work except when the code gets to this line:

train_generator = zip(image_generator, mask_generator)

看来，明显压缩两个列表的过程使它们生成了它们的内容，并且系统开始消耗大量RAM，直到内存用完为止.

it seems the process of zipping the two lists explicitly makes them generate their content and the system starts consuming lots of RAM until it runs out of Memory.

使用Generators的目的是避免在这段代码正好相反的情况下用完RAM.

The point of using Generators is to avoid running out of RAM while this piece of code is exactly doing the opposite.

有什么办法可以解决此问题?

Is there any way to fix this problem?

推荐答案

您可以使用itertools.izip()返回迭代器而不是列表.

You can user itertools.izip() to return an iterator instead of a list.

itertools.izip(*iterables)

Make an iterator that aggregates elements from each of the iterables. Like zip() except that it returns an iterator instead of a list. Used for lock-step iteration over several iterables at a time.

这篇关于Keras-用于图像和蒙版的大型数据集的生成器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！