在多个 GPU 上训练 tensorflow 导致计算机崩溃 [英] Training tensorflow on multiple GPU crashes the computer

问题描述

我们使用以下硬件配置使用 tensorflow 运行多个 GPU 训练:

We use the following hardware configuration to run multiple GPU training using tensorflow:

ubuntu 16.04
cuda 8
cudnn 5.1
8 titan X pascal
220GB of memory

训练代码基于 tensorflow/models github 存储库中发布的 slim.

The training code is based on slim as published in tensorflow/models github repository.

如果我们不使用所有 GPU(最多 4 个，经过测试)，我们就可以运行训练代码.但是，一旦我们使用了所有 8 个 GPU，计算机就会崩溃.

We are able to run the training code if we don't use all GPUs (up to 4, tested). But, once we use all 8 GPUs, the computer crashes.

这可能是什么原因?

推荐答案

我遇到了类似的问题，但对我来说，只要使用一个以上的 GPU，它就会崩溃.对我们来说，修复是将 Linux 内核降级到 2.6.32

I had a similar issue, though for me it crashed as soon as I used more than one GPU. For us, the fix was to downgrade the Linux kernel to 2.6.32

有关我们问题的更多详细信息，请访问:https://groups.google.com/a/tensorflow.org/forum/#!topic/discuss/UjB7uP7_MMU

More details on our problem here: https://groups.google.com/a/tensorflow.org/forum/#!topic/discuss/UjB7uP7_MMU

这篇关于在多个 GPU 上训练 tensorflow 导致计算机崩溃的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！