如何检查keras培训是否已在GPU中运行? [英] How to check if keras training is already running in a GPU?

问题描述

有时我会犯一个错误，并尝试在同一GPU(两个不同的脚本)中使用keras同时运行两次训练，这会使我的机器崩溃或破坏这两次训练.

Sometimes I make a mistake and try to run two simultaneous trainings with keras in the same GPU (two different scripts), making my machine crash or breaking both trainings.

我希望能够在脚本中进行测试，以了解是否正在运行某些培训，因此可以更改gpu或停止新的培训.

I would like to be able to test in my script if there is some training running and therefore either change of gpu or stop the new training.

我发现寻找答案的唯一提示是使用nvidia-smi检查在GPU中运行的进程?

The only hint I found searching for an answer is to use nvidia-smi to check processes running in gpus?

nvidia-smi输出示例:

An example of nvidia-smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 411.63                 Driver Version: 411.63                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp           WDDM  | 00000000:03:00.0 Off |                  N/A |
| 42%   67C    P2    81W / 250W |  10114MiB / 12288MiB |     54%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp           WDDM  | 00000000:04:00.0 Off |                  N/A |
| 35%   58C    P2   144W / 250W |  10315MiB / 12288MiB |     73%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     11660      C   ...\conda\envs\tensorflow18-gpu\python.exe N/A      |
|    1      1532    C+G   Insufficient Permissions                   N/A      |
|    1      5388    C+G   C:\Windows\explorer.exe                    N/A      |
|    1      6648    C+G   Insufficient Permissions                   N/A      |
|    1      7396    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    1      7688    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    1      9808      C   ...\conda\envs\tensorflow18-gpu\python.exe N/A      |
|    1     10820    C+G   Insufficient Permissions                   N/A      |
|    1     11232    C+G   ...x64__8wekyb3d8bbwe\Microsoft.Photos.exe N/A      |
+-----------------------------------------------------------------------------+

在这种情况下，GPU 0和GPU 1中都运行着python.exe.

In this case there is python.exe running in GPU 0 and in GPU 1.

是否有更直接的解决方案?谢谢

Is there a more direct solution? Thanks

推荐答案

您可以尝试以下python软件包， GPUtil

You can try this python package, GPUtil

这篇关于如何检查keras培训是否已在GPU中运行?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！