用于卷积神经网络的float16 vs float32 [英] float16 vs float32 for convolutional neural networks

问题描述

标准是float32，但我想知道在什么情况下可以使用float16?

The standard is float32 but I'm wondering under what conditions it's ok to use float16?

我已经对两种数据类型运行相同的Covnet进行了比较，但没有发现任何问题.对于大型数据集，我更喜欢float16，因为我不必担心内存问题.

I've compared running the same covnet with both datatypes and haven't noticed any issues. With large dataset I prefer float16 because I can worry less about memory issues..

推荐答案

令人惊讶的是，完全可以使用16位，甚至不只是为了娱乐，还可以在生产中使用.例如，在此视频中，杰夫·迪恩(Jeff Dean)谈到了 16位在52:00左右在Google进行计算.幻灯片中的一句话:

Surprisingly, it's totally OK to use 16 bits, even not just for fun, but in production as well. For example, in this video Jeff Dean talks about 16-bit calculations at Google, around 52:00. A quote from the slides:

神经网络训练非常容忍降低精度

Neural net training very tolerant of reduced precision

由于GPU内存是ML计算的主要瓶颈，因此对精度降低进行了大量研究.例如.

Since GPU memory is the main bottleneck in ML computation, there has been a lot of research on precision reduction. E.g.

• 古塔在论文中关于已修复的具有有限数值精度的深度学习" (不是浮动的) 16位训练，但具有随机舍入.

• Gupta at al paper "Deep Learning with Limited Numerical Precision" about fixed (not floating) 16-bit training but with stochastic rounding.

Courbariaux al 以低精度乘法训练深度神经网络" 10位激活和 12位参数更新.

Courbariaux at al "Training Deep Neural Networks with Low Precision Multiplications" about 10-bit activations and 12-bit parameter updates.

这不是限制. Courbariaux等人，"BinaryNet:使用权重和激活力限制为+1或-1来训练深度神经网络" .在这里，他们讨论了 1位的激活和权重(尽管梯度的精度更高)，这使前向通过变得非常快.

And this is not the limit. Courbariaux et al, "BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1". Here they discuss 1-bit activations and weights (though higher precision for the gradients), which makes the forward pass super fast.

当然，我可以想象某些网络可能需要较高的训练精度，但是我建议至少尝试使用16位来训练大型网络，如果事实证明效果较差，则改用32位.

Of course, I can imagine some networks may require high precision for training, but I would recommend at least to try 16 bits for training a big network and switch to 32 bits if it proves to work worse.

这篇关于用于卷积神经网络的float16 vs float32的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！