谷歌colaboratory`ResourceExhaustedError`与GPU [英] google colaboratory `ResourceExhaustedError` with GPU

问题描述

我正在尝试使用colaboratory微调Vgg16模型，但是在使用GPU训练时遇到了此错误.

I'm trying to fine-tune a Vgg16 model using colaboratory but I ran into this error when training with the GPU.

OOM when allocating tensor of shape [7,7,512,4096]

INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor of shape [7,7,512,4096] and type float
     [[Node: vgg_16/fc6/weights/Momentum/Initializer/zeros = Const[_class=["loc:@vgg_16/fc6/weights"], dtype=DT_FLOAT, value=Tensor<type: float shape: [7,7,512,4096] values: [[[0 0 0]]]...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'vgg_16/fc6/weights/Momentum/Initializer/zeros', defined at:

对于我的vm会话也有此输出:

also have this output for my vm session:

    --- colab vm info ---
python v=3.6.3
tensorflow v=1.4.1
tf device=/device:GPU:0
model name  : Intel(R) Xeon(R) CPU @ 2.20GHz
model name  : Intel(R) Xeon(R) CPU @ 2.20GHz
MemTotal:       13341960 kB
MemFree:         1541740 kB
MemAvailable:   10035212 kB

我的tfrecord只有118 256x256张JPG和file size <2MB

My tfrecord is just 118 256x256 JPGs with file size <2MB

有解决方法吗?它在我使用CPU而不是GPU时有效

Is there a workaround? it works when I use the CPU, just not the GPU

推荐答案

看到少量可用的GPU内存几乎总是表明您创建的TensorFlow会话没有allow_growth = True选项.看: https://www.tensorflow.org/guide/using_gpu#allowing_gpu_memory_growth

Seeing a small amount of free GPU memory almost always indicates that you've created a TensorFlow session without the allow_growth = True option. See: https://www.tensorflow.org/guide/using_gpu#allowing_gpu_memory_growth

如果您未设置此选项，则默认情况下，TensorFlow将在创建会话时保留几乎所有GPU内存.

If you don't set this option, by default, TensorFlow will reserve nearly all GPU memory when a session is created.

好消息:从本周开始，Colab现在默认设置此选项，因此当您在Colab上使用多个笔记本时，您应该看到增长率大大降低.而且，您还可以通过在运行时菜单中选择管理会话"来检查每个笔记本的GPU内存使用情况.

Good news: As of this week, Colab now sets this option by default, so you should see much lower growth as you use multiple notebooks on Colab. And, you can also inspect GPU memory usage per notebook by selecting 'Manage session's from the runtime menu.

选择后，您将看到一个对话框，其中列出了所有笔记本电脑以及每个笔记本电脑正在消耗的GPU内存.要释放内存，您也可以从此对话框终止运行时.

Once selected, you'll see a dialog that lists all notebooks and the GPU memory each is consuming. To free memory, you can terminate runtimes from this dialog as well.

这篇关于谷歌colaboratory`ResourceExhaustedError`与GPU的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！