fit_generator 中的 Keras steps_per_epoch 如何工作 [英] How the Keras steps_per_epoch in fit_generator works

问题描述

在 Keras 文档中 - steps_per_epoch: 在宣布一个 epoch 完成并开始下一个 epoch 之前从生成器产生的步骤总数(样本批次).它通常应等于数据集的唯一样本数除以批量大小.

In Keras documentation - steps_per_epoch: Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch. It should typically be equal to the number of unique samples of your dataset divided by the batch size.

我有 3000 个样本.如果我设置了 steps_per_epoch=3000，它的工作会很慢.如果我设置 steps_per_epoch=300 它的工作速度更快，我认为 Batch 工作！

I have 3000 samples. If i set steps_per_epoch=3000 it's work slowly. If i set steps_per_epoch=300 it's work faster and i thought that Batch works!

但后来我比较了在第一种和第二种情况下分配了多少视频内存.并没有注意到很大的不同.如果我使用简单的 fit() 函数，则差异很大.所以这是真正的加速还是我只处理 300 个示例，而不是 3000 个?

But then I compared how much video memory is allocated in the first and second cases. And did not notice a big difference. If I use a simple fit() function then the difference is large. So it's real speed up or i just process 300 examples, instead of 3000?

这个参数需要什么?我怎样才能加快训练速度?我的生成器代码:

What for this parameter is necessary? And how can I speed up the training? My generator code:

def samples_generator(self, path_source, path_mask):
    while 1:
        file_paths_x = self.get_files(path_source)
        file_paths_y = self.get_files(path_mask)
        for path_x, path_y in zip(file_paths_x, file_paths_y):
            x = self.load_pixels(path_x, 3, cv2.INTER_CUBIC)
            y = self.load_pixels(path_y, 0, cv2.INTER_NEAREST)
            yield (x, y)

推荐答案

steps_per_epoch 参数是完成一个完整 epoch 所需的样本批次数.这取决于您的批量大小.批量大小设置在您初始化训练数据的位置.例如，如果您使用 ImageDataGenerator.flow() 或 ImageDataGenerator.flow_from_directory() 执行此操作，则使用 batch_size 指定批次大小> 每个参数中的参数.

The steps_per_epoch parameter is the number of batches of samples it will take to complete one full epoch. This is dependent on your batch size. The batch size is set where you initialize your training data. For example, if you're doing this with ImageDataGenerator.flow() or ImageDataGenerator.flow_from_directory(), the batch size is specified with the batch_size parameter in each of these.

你说你有 3000 个样本.

You said you have 3000 samples.

• 如果您的批次大小为 100，则 steps_per_epoch 将为 30.
• 如果您的批次大小为 10，则 steps_per_epoch 将为 300.
• 如果您的批次大小为 1，则 steps_per_epoch 将为 3000.

• If your batch size was 100, then steps_per_epoch would be 30.
• If your batch size was 10, then steps_per_epoch would be 300.
• If your batch size was 1, then steps_per_epoch would be 3000.

这是因为steps_per_epoch 应该等于样本总数除以批量大小.在下面的两个视频中提供了在 Keras 中实现此功能的过程.

This is because steps_per_epoch should be equivalent to the total number of samples divided by the batch size. The process of implementing this in Keras is available in the two videos below.

必须设置 steps_per_epoch 的原因是生成器旨在无限期运行(请参阅 文档:

The reason why you have to set steps_per_epoch is that the generator is designed to run indefinitely (See the docs:

预计生成器将无限期循环其数据."

"The generator is expected to loop over its data indefinitely."

).您通过设置 while 1 来实现这一点.由于 fit_generator() 应该运行 epochs=x 次，所以该方法必须知道在这个无限循环中的下一个 epoch 何时开始(因此，数据必须是又从头画了).

). You implemented this by setting while 1. Since fit_generator() is supposed to run epochs=x times, the method must know when the next epoch begins within this indefinitely loop (and, hence, the data has to be drawn from the beginning again).

• 使用 Keras 进行 CNN 训练的图像准备
• 在 Keras 中创建和训练 CNN

这篇关于fit_generator 中的 Keras steps_per_epoch 如何工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！