有没有办法确定TensorFlow使用了多少GPU内存? [英] Is there a way of determining how much GPU memory is in use by TensorFlow?

问题描述

Tensorflow倾向于在其GPU上预分配全部可用内存.对于调试，是否有办法知道实际使用了多少内存?

Tensorflow tends to preallocate the entire available memory on it's GPUs. For debugging, is there a way of telling how much of that memory is actually in use?

推荐答案

(1)

(1) There is some limited support with Timeline for logging memory allocations. Here is an example for its usage:

    run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()
    summary, _ = sess.run([merged, train_step],
                          feed_dict=feed_dict(True),
                          options=run_options,
                          run_metadata=run_metadata)
    train_writer.add_run_metadata(run_metadata, 'step%03d' % i)
    train_writer.add_summary(summary, i)
    print('Adding run metadata for', i)
    tl = timeline.Timeline(run_metadata.step_stats)
    print(tl.generate_chrome_trace_format(show_memory=True))
    trace_file = tf.gfile.Open(name='timeline', mode='w')
    trace_file.write(tl.generate_chrome_trace_format(show_memory=True))

您可以尝试使用MNIST示例(带有摘要的摘要)

You can give this code a try with the MNIST example (mnist with summaries)

这将生成一个名为时间轴的跟踪文件，您可以使用chrome://tracing打开该文件.请注意，这仅提供了近似的GPU内存使用情况统计信息.它基本上模拟了GPU执行，但是无法访问完整的图形元数据.它也不知道已经为GPU分配了多少个变量.

This will generate a tracing file named timeline, which you can open with chrome://tracing. Note that this only gives an approximated GPU memory usage statistics. It basically simulated a GPU execution, but doesn't have access to the full graph metadata. It also can't know how many variables have been assigned to the GPU.

(2)为了对GPU内存使用情况进行非常粗略的衡量，nvidia-smi将在您运行命令时显示设备的总内存使用情况.

(2) For a very coarse measure of GPU memory usage, nvidia-smi will show the total device memory usage at the time you run the command.

nvprof可以在CUDA内核级别显示片上共享内存使用情况和寄存器使用情况，但不显示全局/设备内存使用情况.

nvprof can show the on-chip shared memory usage and register usage at the CUDA kernel level, but doesn't show the global/device memory usage.

这是一个示例命令:nvprof --print-gpu-trace matrixMul

Here is an example command: nvprof --print-gpu-trace matrixMul

此处有更多详细信息: http://docs.nvidia.com/cuda/profiler-users-guide/#abstract

And more details here: http://docs.nvidia.com/cuda/profiler-users-guide/#abstract

这篇关于有没有办法确定TensorFlow使用了多少GPU内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！