如何分析Python脚本的CPU使用率? [英] How to profile CPU usage of a Python script?

问题描述

理想情况下，我想要记录执行深层神经网络 Keras 模型的Python脚本的CPU使用情况.我正在寻找与 memory_profiler 等效的CPU，它提供了进程的内存消耗. /p>

我已经研究过使用 psutil (建议在此高​​级答案建议 cProfile给出脚本中函数的运行时间.尽管这并不是我想要的，但我注意到它返回了 CPU的总秒数，至少可以让我比较这方面的CPU使用率.

解决方案

您可以在子进程中运行model.predict(x_test)，并在主进程中同时记录其CPU使用率.例如，

import time
import multiprocessing as mp
import psutil
import numpy as np
from keras.models import load_model

def run_predict():
    model = load_model('1.h5')
    x_test = np.random.rand(10000, 1000)
    time.sleep(1)

    for _ in range(3):
        model.predict(x_test)
        time.sleep(0.5)

def monitor(target):
    worker_process = mp.Process(target=target)
    worker_process.start()
    p = psutil.Process(worker_process.pid)

    # log cpu usage of `worker_process` every 10 ms
    cpu_percents = []
    while worker_process.is_alive():
        cpu_percents.append(p.cpu_percent())
        time.sleep(0.01)

    worker_process.join()
    return cpu_percents

cpu_percents = monitor(target=run_predict)

上述脚本的cpu_percents中的值类似于:

Ideally what I want is to record the CPU usage of a Python script that is executing a deep neural net Keras model. I'm looking for the CPU equivalent of memory_profiler, which provides the memory consumption of a process.

I have looked at using psutil (suggested in this answer) which would indicate my script could contain some variant of

p = psutil.Process(current_pid)
p.cpu_percent()

but the problem is the important function call I really want to capture the effect of would be the inference stage of the model

model.predict(x_test)

and if I run psutil before/after this step the CPU usage recorded won't be a true reflection of the CPU usage of the process.

So then I was thinking could I use something like top/htop to log the CPU usage of the script to some file, capturing the fluctuating CPU usage while the process is running, and then calculate an average (or something similar) after the fact. The issue I see with this, however, is don't I need to know the PID to utilise top, so how can I use top to monitor the script before it is executed (and hasn't even been assigned a PID)?

I can see this highly-ranked answer suggests cProfile which gives the running time of functions within a script. Although this isn't exactly what I want I do notice that it returns the total number of CPU seconds, which would at least let me compare CPU usage in that regard.

解决方案

You can run model.predict(x_test) in a subprocess and log its CPU usage simultaneously in the main process. For example,

import time
import multiprocessing as mp
import psutil
import numpy as np
from keras.models import load_model

def run_predict():
    model = load_model('1.h5')
    x_test = np.random.rand(10000, 1000)
    time.sleep(1)

    for _ in range(3):
        model.predict(x_test)
        time.sleep(0.5)

def monitor(target):
    worker_process = mp.Process(target=target)
    worker_process.start()
    p = psutil.Process(worker_process.pid)

    # log cpu usage of `worker_process` every 10 ms
    cpu_percents = []
    while worker_process.is_alive():
        cpu_percents.append(p.cpu_percent())
        time.sleep(0.01)

    worker_process.join()
    return cpu_percents

cpu_percents = monitor(target=run_predict)

The values in cpu_percents for the above script would be something like:

这篇关于如何分析Python脚本的CPU使用率?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！