如何在 Python 中获取每秒的 GPU 使用率 [英] How to get every second's GPU usage in Python
问题描述
我有一个由 tensorflow-gpu
运行的模型,我的设备是 nvidia
.我想列出每一秒的 GPU 使用率,以便我可以测量平均/最大 GPU 使用率.我可以手动打开两个终端,一个是运行模型,另一个是通过nvidia-smi -l 1
测量.当然,这不是一个好方法.我也尝试使用 Thread
来做到这一点,在这里.
I have a model which runs by tensorflow-gpu
and my device is nvidia
. And I want to list every second's GPU usage so that I can measure average/max GPU usage. I can do this mannually by open two terminals, one is to run model and another is to measure by nvidia-smi -l 1
. Of course, this is not a good way. I also tried to use a Thread
to do that, here it is.
import subprocess as sp
import os
from threading import Thread
class MyThread(Thread):
def __init__(self, func, args):
super(MyThread, self).__init__()
self.func = func
self.args = args
def run(self):
self.result = self.func(*self.args)
def get_result(self):
return self.result
def get_gpu_memory():
output_to_list = lambda x: x.decode('ascii').split('\n')[:-1]
ACCEPTABLE_AVAILABLE_MEMORY = 1024
COMMAND = "nvidia-smi -l 1 --query-gpu=memory.used --format=csv"
memory_use_info = output_to_list(sp.check_output(COMMAND.split()))[1:]
memory_use_values = [int(x.split()[0]) for i, x in enumerate(memory_use_info)]
return memory_use_values
def run():
pass
t1 = MyThread(run, args=())
t2 = MyThread(get_gpu_memory, args=())
t1.start()
t2.start()
t1.join()
t2.join()
res1 = t2.get_result()
然而,这也不会返回每一秒的使用情况.有什么好的解决办法吗?
However, this does not return every second's usage as well. Is there a good solution?
推荐答案
在命令nvidia-smi -l 1 --query-gpu=memory.used --format=csv
-l 代表:
-l, --loop= 以指定的第二个间隔探测直到 Ctrl+C.
所以命令:
COMMAND = 'nvidia-smi -l 1 --query-gpu=memory.used --format=csv'
sp.check_output(COMMAND.split())
永远不会终止并返回.
如果您将事件循环从命令 (nvidia-smi) 中删除到 python,它会起作用.
It works if you remove the event loop from the command(nvidia-smi) to python.
代码如下:
import subprocess as sp
import os
from threading import Thread , Timer
import sched, time
def get_gpu_memory():
output_to_list = lambda x: x.decode('ascii').split('\n')[:-1]
ACCEPTABLE_AVAILABLE_MEMORY = 1024
COMMAND = "nvidia-smi --query-gpu=memory.used --format=csv"
try:
memory_use_info = output_to_list(sp.check_output(COMMAND.split(),stderr=sp.STDOUT))[1:]
except sp.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
memory_use_values = [int(x.split()[0]) for i, x in enumerate(memory_use_info)]
# print(memory_use_values)
return memory_use_values
def print_gpu_memory_every_5secs():
"""
This function calls itself every 5 secs and print the gpu_memory.
"""
Timer(5.0, print_gpu_memory_every_5secs).start()
print(get_gpu_memory())
print_gpu_memory_every_5secs()
"""
Do stuff.
"""
这篇关于如何在 Python 中获取每秒的 GPU 使用率的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!