TF 准确度得分和混淆矩阵不一致.TensorFlow 是否在每次访问 BatchDataset 时对数据进行混洗? [英] TF accuracy score and confusion matrix disagree. Is TensorFlow shuffling data on each access of BatchDataset?

问题描述

model.evaluate() 报告的准确度与根据 Sklearn 或 TF 混淆矩阵计算的准确度有很大不同.

Accuracy reported by model.evaluate() is very different from accuracy calculated from Sklearn or TF confusion matrix.

from sklearn.metrics import confusion_matrix
...

training_data, validation_data, testing_data = load_img_datasets()
# These ^ are tensorflow.python.data.ops.dataset_ops.BatchDataset

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = create_model(INPUT_SHAPE, NUM_CATEGORIES)
    optimizer = tf.keras.optimizers.Adam()
    metrics = ['accuracy']
    model.compile(loss='categorical_crossentropy',
                  optimizer=optimizer,
                  metrics=metrics)

history = model.fit(training_data, epochs=epochs,
                    validation_data=validation_data)

testing_data.shuffle(len(testing_data), reshuffle_each_iteration=False)
# I think this ^ is preventing additional shuffles on access

loss, accuracy = model.evaluate(testing_data)
print(f"Accuracy: {(accuracy * 100):.2f}%")
# Prints 
# Accuracy: 78.7%

y_hat = model.predict(testing_data)
y_test = np.concatenate([y for x, y in testing_data], axis=0)
c_matrix = confusion_matrix(np.argmax(y_test, axis=-1),
                            np.argmax(y_hat, axis=-1))
print(c_matrix)
# Prints result that does not agree:
# Confusion matrix:
#[[ 72 111  54  15  69]
# [ 82 100  44  16  78]
# [ 64 114  52  21  69]
# [ 71 106  54  21  68]
# [ 79 101  51  25  64]]
# Accuracy calculated from CM = 19.3%

一开始，我以为 TensorFlow 在每次访问时都对 testing_data 进行了混洗，所以我添加了 testing_data.shuffle(len(testing_data), reshuffle_each_iteration=False)，但还是有结果不同意.

At first, I thought that TensorFlow was shuffling testing_data on each access so I added testing_data.shuffle(len(testing_data), reshuffle_each_iteration=False), but still results do not agree.

也尝试过 TF 混淆矩阵:

Have also tried TF confusion matrix:

y_hat = model.predict(testing_data)
y_test = np.concatenate([y for x, y in testing_data], axis=0)
true_class = tf.argmax(y_test, 1)
predicted_class = tf.argmax(y_hat, 1)
cm = tf.math.confusion_matrix(true_class, predicted_class, NUM_CATEGORIES)
print(cm)

...具有相似的结果.

...with similar result.

显然预测的标签必须与正确的标签进行比较.我做错了什么?

Obviously predicted labels must be compared with the correct labels. What am I doing wrong?

推荐答案

我找不到源代码，但似乎 Tensorflow 仍在幕后改组测试.您可以尝试遍历数据集以获得预测和真实类别:

I could not find the source but seems like Tensorflow is still shuffling the testing under the hood. You can try to iterate over the dataset to obtain predictions and real classes:

predicted_classes = np.array([])
true_classes =  np.array([])

for x, y in testing_data:
  predicted_classes = np.concatenate([predicted_classes,
                       np.argmax(model(x), axis = -1)])
  true_classes = np.concatenate([true_classes, np.argmax(y.numpy(), axis=-1)])

model(x) 用于更快的执行.来自源码:

计算是分批完成的.这种方法是专为表现在大规模投入.对于适合一批的少量输入，建议直接使用 __call__ 以加快执行速度，例如，model(x)

Computation is done in batches. This method is designed for performance in large scale inputs. For small amount of inputs that fit in one batch, directly using __call__ is recommended for faster execution, e.g., model(x)

如果不行，你可以试试model.predict(x).

If it does not work, you can try model.predict(x) instead.

这篇关于TF 准确度得分和混淆矩阵不一致.TensorFlow 是否在每次访问 BatchDataset 时对数据进行混洗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！