分割概率图取决于 CPU 或 GPU [英] Segmentation probablity maps depend on CPU or GPU

问题描述

我在 tensorflow 中写了一个语义分割模型.该模型在 CPU 中训练，预测也在 CPU 中完成.一段时间后，我在 GPU 机器上再次运行预测，我注意到概率图中存在一些非常小的差异(附图).另外，我想检查一下batch size是否影响了预测分布(我一开始以为没有影响).

我做了一个小实验来描述这四种情况:

A1) 在 CPU 中训练的模型和在 CPU 中运行的预测 -->批量大小 = 1

A2) 在 CPU 中训练的模型和在 CPU 中运行的预测 -->批量大小 = 64

B1) 在 CPU 中训练的模型和在 GPU 中运行的预测 -->批量大小 = 1

B2) 在 CPU 中训练的模型和在 GPU 中运行的预测 -->批量大小 = 64

我已经总结了附图中测试集中其中一张图像的一部分的结果.

我们可以观察到:

1. 训练模型并在 CPU 中运行预测提供相同的结果，无论批量大小如何.
2. 在 CPU 中训练模型并在 GPU 中运行预测时，批量大小会产生影响(尽管很小)
3. CPU 和 GPU 提供的预测不匹配(差异非常小).如果截止阈值设置为 0.55，那么这可能是相关的，然后根据附图，CPU 将其分类为背景像素(0.549857＜0.55)，而 GPU 将其分类为前景像素(0.550076＞0.550076).0.55) 如果图像是更大批次的一部分.
4. 再次运行 GPU 预测，提供不同的概率(再次出现非常小的差异)，而 CPU 始终提供相同的概率.

这是用于重现该功能的

解决方案

这个问题发布在 Keras 的 github 并得到了 Keras 团队的相应答复.

可以提出以下关键点:

1. 在 GPU 中运行模型会产生不确定的结果(每次运行模型时，它都会提供不同的输出，即使差异非常小
2. CPU 产生确定性的结果(总是相同的结果)
3. 有计划使 GPU 预测具有确定性，但它们仍与 CPU 结果略有不同.
4. 在 TF>2.5 中使用环境变量 TF_DETERMINISTIC_OPS=1 有助于解决不确定性行为.

I wrote a model for semantic segmentation in tensorflow. The model were trained in a CPU and the predictions were also done in a CPU. After some time I run the predictions again in a GPU machine and I noticed that there were some very small differences in the probability maps (Figure attached). In addition, I wanted to check if the batch size affected the prediction distribution (I initially thought that there were no influence).

I ran a small experiment describing these four situations:

A1) Model Trained in CPU and Predictions run in CPU --> Batch Size = 1

A2) Model Trained in CPU and Prediction run in CPU --> Batch Size = 64

B1) Model Trained in CPU and Predictions run in GPU --> Batch Size = 1

B2) Model Trained in CPU and Prediction run in GPU --> Batch Size = 64

I have summarized the results for a portion of one of the images in the test set in the Figure attached.

We can observe that:

1. Training the model and running the predictions in a CPU provides the same results regardless the batch size.
2. Batch size has an effect (although small) when training the model in a CPU and running the predictions in a GPU
3. Predictions provided by a CPU and a GPU don't match (very small differences). This can be relevant if the cutoff threshold is placed at 0.55, then according to the Figure attached, the CPU will classify it as a background pixel (0.549857<0.55) , whereas the GPU will classify it as a foreground pixel (0.550076>0.55) if the image was part of a larger batch.
4. Running the GPU predictions again, provide different probabilities (very small differences again), whereas CPU provides always the same probabilities.

Here is the link to the colab to reproduce the feature.

So my main question is:

• Is this behaviour something normal that is expected to happen?
• Why this behavior happens (documentation)?
• How can I avoid this to happen? So that the Predictions run in a GPU always provide the same probabilities as the CPU) and regardless of the batch size?

解决方案

This question was posted in Keras' github and was answered accordingly by Keras' team.

The following keypoints could be made:

1. Running a model in GPU produces non-deterministic results (it provides a different output, even with really small differences, every time the model is run
2. CPU produces deterministic results (always the same results)
3. There are plans to make GPU predictions deterministic, but they still be slightly different to the CPU results.
4. Using the environmental variable TF_DETERMINISTIC_OPS=1 in TF>2.5 helps with the non-deterministic behavior.

这篇关于分割概率图取决于 CPU 或 GPU的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！