GPU中的Bfloat16培训 [英] Bfloat16 training in GPUs

问题描述

我正在尝试使用新的bfloat16数据类型变量训练模型。我知道Google TPU支持此功能。我想知道是否有人尝试使用GPU进行训练（例如GTX 1080 Ti）。 GPU张量内核是否具有支持性，这是否有可能？如果有人有任何经验，请分享您的想法。
非常感谢！

Hi I am trying to train a model using the new bfloat16 datatype variables. I know this is supported in Google TPUs. I was wondering if anyone has tried training using GPUs (for example, GTX 1080 Ti). Is that even possible, whether the GPU tensor cores are supportive? If anyone has any experience please share your thoughts. Many thanks!

推荐答案

我已在Tensorflow github社区中发布了此问题。到目前为止，这是他们的回应-

bfloat16对GPU的支持不完整，因为设备本身不支持。

I had posted this question in Tensorflow github community. Here is their response so far - " bfloat16 support isn't complete for GPUs, as it's not supported natively by the devices.

您需要使用float32或float16来执行GPU性能（尽管float16可能很难训练模型）。TPU支持bfloat16来有效地进行所有操作（但是您当前必须迁移模型才能在TPU上工作）。 $ b

For performance you'll want to use float32 or float16 for GPU execution (though float16 can be difficult to train models with). TPUs support bfloat16 for effectively all operations (but you currently have to migrate your model to work on the TPU). "

这篇关于GPU中的Bfloat16培训的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！