Tensorflow 因 CUBLAS_STATUS_ALLOC_FAILED 崩溃 [英] Tensorflow crashes with CUBLAS_STATUS_ALLOC_FAILED

问题描述

我正在使用简单的 MINST 神经网络程序在 Windows 10 上运行 tensorflow-gpu.当它尝试运行时，遇到 CUBLAS_STATUS_ALLOC_FAILED 错误.谷歌搜索没有发现任何东西.

I'm running tensorflow-gpu on Windows 10 using a simple MINST neural network program. When it tries to run, it encounters a CUBLAS_STATUS_ALLOC_FAILED error. A google search doesn't turn up anything.

I c:	f_jenkinshomeworkspace
elease-windevicegpuoswindows	ensorflowcorecommon_runtimegpugpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 970
major: 5 minor: 2 memoryClockRate (GHz) 1.253
pciBusID 0000:0f:00.0
Total memory: 4.00GiB
Free memory: 3.31GiB
I c:	f_jenkinshomeworkspace
elease-windevicegpuoswindows	ensorflowcorecommon_runtimegpugpu_device.cc:906] DMA: 0
I c:	f_jenkinshomeworkspace
elease-windevicegpuoswindows	ensorflowcorecommon_runtimegpugpu_device.cc:916] 0:   Y
I c:	f_jenkinshomeworkspace
elease-windevicegpuoswindows	ensorflowcorecommon_runtimegpugpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 970, pci bus id: 0000:0f:00.0)
E c:	f_jenkinshomeworkspace
elease-windevicegpuoswindows	ensorflowstream_executorcudacuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
W c:	f_jenkinshomeworkspace
elease-windevicegpuoswindows	ensorflowstream_executorstream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
  File "C:UsersAnonymousAppDataLocalProgramsPythonPython35libsite-packages	ensorflowpythonclientsession.py", line 1021, in _do_call
    return fn(*args)
  File "C:UsersAnonymousAppDataLocalProgramsPythonPython35libsite-packages	ensorflowpythonclientsession.py", line 1003, in _run_fn
    status, run_metadata)
  File "C:UsersAnonymousAppDataLocalProgramsPythonPython35libcontextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:UsersAnonymousAppDataLocalProgramsPythonPython35libsite-packages	ensorflowpythonframeworkerrors_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : a.shape=(100, 784), b.shape=(784, 256), m=100, n=256, k=784
         [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](_recv_Placeholder_0/_7, Variable/read)]]
         [[Node: Mean/_15 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_35_Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

推荐答案

allow_growth"的位置会话配置的属性现在似乎有所不同.这里有解释:https://www.tensorflow.org/tutorials/using_gpu

The location of the "allow_growth" property of the session config seems to be different now. It's explained here: https://www.tensorflow.org/tutorials/using_gpu

所以目前你必须这样设置:

So currently you'd have to set it like this:

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)

这篇关于Tensorflow 因 CUBLAS_STATUS_ALLOC_FAILED 崩溃的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！