fit_generator中的Keras steps_per_epoch如何工作 [英] How the Keras steps_per_epoch in fit_generator works

问题描述

在Keras文档中- steps_per_epoch :在声明一个纪元完成并开始下一个纪元之前，要从生成器产生的总步数(批次).通常应等于数据集中唯一样本的数量除以批次大小.

In Keras documentation - steps_per_epoch: Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch. It should typically be equal to the number of unique samples of your dataset divided by the batch size.

我有3000个样本. 如果我将steps_per_epoch = 3000设置为慢速.如果我将steps_per_epoch = 300设置为更快，那么我认为批处理有效！

I have 3000 samples. If i set steps_per_epoch=3000 it's work slowly. If i set steps_per_epoch=300 it's work faster and i thought that Batch works!

但是随后我比较了在第一种和第二种情况下分配了多少视频内存.并没有注意到有很大的不同.如果我使用简单的 fit()函数，则差异很大.所以这是真正的加速，还是我只处理300个示例，而不是3000个示例?

But then I compared how much video memory is allocated in the first and second cases. And did not notice a big difference. If I use a simple fit() function then the difference is large. So it's real speed up or i just process 300 examples, instead of 3000?

该参数需要什么?我如何加快培训速度? 我的生成器代码:

What for this parameter is necessary? And how can I speed up the training? My generator code:

def samples_generator(self, path_source, path_mask):
    while 1:
        file_paths_x = self.get_files(path_source)
        file_paths_y = self.get_files(path_mask)
        for path_x, path_y in zip(file_paths_x, file_paths_y):
            x = self.load_pixels(path_x, 3, cv2.INTER_CUBIC)
            y = self.load_pixels(path_y, 0, cv2.INTER_NEAREST)
            yield (x, y)

推荐答案

steps_per_epoch参数是完成一个完整纪元所需的样品批数.这取决于您的批次大小.在您初始化训练数据的地方设置批次大小.例如，如果您使用ImageDataGenerator.flow()或ImageDataGenerator.flow_from_directory()进行此操作，则批处理大小将在每个参数中均使用batch_size参数指定.

The steps_per_epoch parameter is the number of batches of samples it will take to complete one full epoch. This is dependent on your batch size. The batch size is set where you initialize your training data. For example, if you're doing this with ImageDataGenerator.flow() or ImageDataGenerator.flow_from_directory(), the batch size is specified with the batch_size parameter in each of these.

您说您有3000个样本.

You said you have 3000 samples.

• 如果批次大小为100，则steps_per_epoch为30.
• 如果批次大小为10，则steps_per_epoch为300.
• 如果批次大小为1，则steps_per_epoch为3000.

• If your batch size was 100, then steps_per_epoch would be 30.
• If your batch size was 10, then steps_per_epoch would be 300.
• If your batch size was 1, then steps_per_epoch would be 3000.

这是因为steps_per_epoch应等于样品总数除以批次大小.下面的两个视频中提供了在Keras中实现此过程的过程.

This is because steps_per_epoch should be equivalent to the total number of samples divided by the batch size. The process of implementing this in Keras is available in the two videos below.

必须设置steps_per_epoch的原因是生成器被设计为可无限期运行(请参见

The reason why you have to set steps_per_epoch is that the generator is designed to run indefinitely (See the docs:

预计生成器将无限期地循环其数据."

"The generator is expected to loop over its data indefinitely."

).您可以通过设置while 1来实现. 由于fit_generator()应该运行epochs=x次，因此该方法必须知道下一个纪元何时在此无限循环内开始(因此，必须从头开始绘制数据).

). You implemented this by setting while 1. Since fit_generator() is supposed to run epochs=x times, the method must know when the next epoch begins within this indefinitely loop (and, hence, the data has to be drawn from the beginning again).

• 使用Keras进行CNN训练的图像准备
• 在Keras中创建和训练CNN

• Image preparation for CNN training with Keras
• Create and train a CNN in Keras

这篇关于fit_generator中的Keras steps_per_epoch如何工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！