称"fit"为"fit".多次在Keras [英] Calling "fit" multiple times in Keras

问题描述

我正在研究CNN上超过数百GB的图像.我创建了一个训练功能，该功能可以消除这些图像的4Gb块，并在每个片段上调用fit.我担心我只训练最后一块而不是整个数据集.

I've working on a CNN over several hundred GBs of images. I've created a training function that bites off 4Gb chunks of these images and calls fit over each of these pieces. I'm worried that I'm only training on the last piece on not the entire dataset.

有效地，我的伪代码如下:

Effectively, my pseudo-code looks like this:

DS = lazy_load_400GB_Dataset()
for section in DS:
    X_train = section.images
    Y_train = section.classes

    model.fit(X_train, Y_train, batch_size=16, nb_epoch=30)

我知道API和Keras论坛都说它将在整个数据集上进行训练，但是我无法直观地理解为什么网络不会仅在最后一个训练块上进行重新学习.

I know that the API and the Keras forums say that this will train over the entire dataset, but I can't intuitively understand why the network wouldn't relearn over just the last training chunk.

一些帮助理解这一点将不胜感激.

Some help understanding this would be much appreciated.

最好， 乔

推荐答案

对于不适合内存的数据集，

For datasets that do not fit into memory, there is an answer in the Keras Documentation FAQ section

您可以使用model.train_on_batch(X, y)和 model.test_on_batch(X, y).请参见模型文档.

You can do batch training using model.train_on_batch(X, y) and model.test_on_batch(X, y). See the models documentation.

或者，您可以编写一个生成批量生成器的生成器. 训练数据并使用方法model.fit_generator(data_generator, samples_per_epoch, nb_epoch).

Alternatively, you can write a generator that yields batches of training data and use the method model.fit_generator(data_generator, samples_per_epoch, nb_epoch).

您可以在我们的 CIFAR10示例.

因此，如果您要按照自己的方式迭代数据集，则可能应该使用model.train_on_batch并自己照顾批处理大小和迭代.

So if you want to iterate your dataset the way you are doing, you should probably use model.train_on_batch and take care of the batch sizes and iteration yourself.

还有一点要注意的是，您应该确保在每个时期之后对训练模型所用的样本进行排序.您编写示例代码的方式似乎并没有改组数据集.您可以在此处和此处

One more thing to note is that you should make sure the order in which the samples you train your model with is shuffled after each epoch. The way you have written the example code seems to not shuffle the dataset. You can read a bit more about shuffling here and here

这篇关于称"fit"为"fit".多次在Keras的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！