valuate_generator的准确度为0％，但是在使用相同数据进行训练期间的准确度为75％-发生了什么? [英] 0% accuracy with evaluate_generator but 75% accuracy during training with same data - what is going on?

问题描述

使用ImageDataGenerator，fit_generator和valuate_generator的keras模型遇到了非常奇怪的事情.

I'm encountering a very strange with a keras model using ImageDataGenerator, fit_generator, and evaluate_generator.

我正在像这样创建模型:

I'm creating the model like so:

classes = <list of classes>
num_classes = len(classes)

pretrained_model = Sequential()
pretrained_model.add(ResNet50(include_top=False, weights='imagenet', pooling='avg'))
pretrained_model.add(Dense(num_classes, activation='softmax'))

pretrained_model.layers[0].trainable = False

pretrained_model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

我正在这样训练它:

idg_final = ImageDataGenerator(
    data_format='channels_last',
    rescale=1./255,
    width_shift_range = 0.2,
    height_shift_range = 0.2,
    rotation_range=15,
)

traing_gen = idg_final.flow_from_directory('./train', classes=classes, target_size=(224, 224), class_mode='categorical')

pretrained_model.fit_generator(traing_gen, epochs=1, verbose=1)

fit_generator打印loss: 1.0297 - acc: 0.7546.

然后，我正在尝试使用与训练时完全相同的数据评估模型.

Then, I am trying to evaluate the model on the exact same data it was trained on.

debug_gen = idg_final.flow_from_directory('./train', target_size=(224, 224), class_mode='categorical', classes=classes, shuffle=True)
print(pretrained_model.evaluate_generator(debug_gen, steps=100))

哪个打印[10.278913383483888, 0.0].

为什么在相同的精确数据上准确性如此不同?

Why is the accuracy so different on the same exact data?

我还想指出，有时精度高于0.0.例如，当我使用经过五个时期训练的模型时，evaluate_accuracy返回的精度为6％.

I also wanted to point out that sometimes the accuracy is above 0.0. For example, when I use a model trained with five epochs, evaluate_accuracy returns 6% accuracy.

根据以下答案，我确保训练更多的时期，并且ImageDataGenerator进行评估时没有随机的移位和旋转.在同一数据集上的训练过程中，我仍然获得很高的准确性，而在评估过程中，我得到的准确性却非常低.

Edit 2: Based on the answers below I made sure to train for more epochs and that the ImageDataGenerator for evaluation did not have random shifts and rotations. I'm still getting very high accuracy during training and extremely low accuracy during evaluation on the same dataset.

我的训练方式

idg_final = ImageDataGenerator(
    data_format='channels_last',
    rescale=1./255,
    width_shift_range = 0.2,
    height_shift_range = 0.2,
    rotation_range=15,
)

traing_gen = idg_final.flow_from_directory('./train', classes=classes, target_size=(224, 224), class_mode='categorical')                  

pretrained_model.fit_generator(traing_gen, epochs=10, verbose=1)

打印以下内容:

Found 9850 images belonging to 4251 classes.
Epoch 1/10
308/308 [==============================] - 3985s 13s/step - loss: 8.9218 - acc: 0.0860
Epoch 2/10
308/308 [==============================] - 3555s 12s/step - loss: 3.2710 - acc: 0.3403
Epoch 3/10
308/308 [==============================] - 3594s 12s/step - loss: 1.8597 - acc: 0.5836
Epoch 4/10
308/308 [==============================] - 3656s 12s/step - loss: 1.2712 - acc: 0.7058
Epoch 5/10
308/308 [==============================] - 3667s 12s/step - loss: 0.9556 - acc: 0.7795
Epoch 6/10
308/308 [==============================] - 3689s 12s/step - loss: 0.7665 - acc: 0.8207
Epoch 7/10
308/308 [==============================] - 3693s 12s/step - loss: 0.6581 - acc: 0.8498
Epoch 8/10
308/308 [==============================] - 3618s 12s/step - loss: 0.5874 - acc: 0.8636
Epoch 9/10
308/308 [==============================] - 3823s 12s/step - loss: 0.5144 - acc: 0.8797
Epoch 10/10
308/308 [==============================] - 4334s 14s/step - loss: 0.4835 - acc: 0.8854

我正在完全相同的数据集

idg_debug = ImageDataGenerator(
    data_format='channels_last',
    rescale=1./255,
)

debug_gen = idg_debug.flow_from_directory('./train', target_size=(224, 224), class_mode='categorical', classes=classes)
print(pretrained_model.evaluate_generator(debug_gen))

打印以下非常低的精度:[10.743386410747084, 0.0001015228426395939]

Which prints the following very low accuracy: [10.743386410747084, 0.0001015228426395939]

完整代码为此处.

推荐答案

我怀疑有两件事.

1-不，您的数据不相同.

您在ImageDataGenerator中使用了三种类型的扩充，似乎没有设置随机种子.因此，测试数据不等于训练数据.

You're using three types of augmentation in ImageDataGenerator, and it seems there isn't a random seed being set. So, test data is not equal to training data.

而且看来，您也只训练了一个时期，这非常少(除非您确实有大量数据，但是由于您使用的是增强，也许并非如此). (PS:我在您的fit_generator通话中没有看到steps_per_epoch参数...)

And as it seems, you're also training for only one epoch, which is very little (unless you really have tons of data, but since you're using augmentation, maybe that's not the case). (PS: I don't see the steps_per_epoch argument in your fit_generator call...)

因此，如果您想获得良好的效果，请参考以下解决方案:

So, if you want to see good results, here are some solutions:

• 从生成器中删除此测试的扩充参数(训练和测试数据)-这意味着删除width_shift_range，height_shift_range和rotation_range；
• 如果没有，请花很长时间训练，足以使您的模型真正适应各种增强图像(看来，五个时期似乎仍然太少了)；
• 或设置随机种子并确保测试数据等于训练数据(flow_from_directory中的参数seed)

• remove the augmentation arguments from the generator for this test (either training and test data) - This means, remove width_shift_range, height_shift_range and rotation_range;
• if not, train for really long, enough for your model to really get used to all kinds of augmented images (as it seems, five epochs seem still to be way too little);
• or set a random seed and guarantee that the test data is equal to the training data (argument seed in flow_from_directory)

2-(如果您不是Keras/编程的新手，可能会发生这种情况，因此请忽略这种情况).您可能正在运行再次定义模型的代码在测试时.

2 - (This may happen if you're very new to Keras/programming, so please ignore if it's not the case) You might be running the code that defines the model again when testing.

如果您再次运行定义模型的代码，它将用随机权重替换您之前的所有训练.

If you run the code that defines the model again, it will replace all your previous training with random weights.

3-由于我们没有建议:

也许保存权重而不是保存模型.我通常这样做而不是保存模型. (出于某种原因，我不明白，我从未能够加载过这样的模型)

Maybe save the weights instead of saving the model. I usually do this instead of saving the models. (For some reason I don't understand, I've never been able to load a model like that)

def createModel():
    ....

model = createModel()
...
model.fit_generator(....)

np.save('model_weights.npy',model.get_weights())

model = createModel()
model.set_weights(np.load('model_weights.npy'))
...
model.evaluate_generator(...)

---

提示:

它与bug无关，但请确保基本模型层确实是第0层.如果我还记得的话，顺序模型有一个输入层，而实际上应该使第1层不可训练.

It's not related to the bug, but make sure that the base model layer is really layer 0. If I remember well, sequential models have an input layer and you should actually be making layer 1 untrainable instead.

使用model.summary()确认不可训练参数的数量.

Use the model.summary() to confirm the number of untrainable parameters.

这篇关于valuate_generator的准确度为0％，但是在使用相同数据进行训练期间的准确度为75％-发生了什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！