通过 Python 函数跟踪*最大*内存使用情况 [英] Tracking *maximum* memory usage by a Python function

问题描述

我想知道在调用函数期间分配的最大 RAM 量是多少(在 Python 中).还有其他与跟踪 RAM 使用情况相关的 SO 问题:

I want to find out what the maximum amount of RAM allocated during the call to a function is (in Python). There are other questions on SO related to tracking RAM usage:

推荐使用哪种 Python 内存分析器?

如何在 Python 中分析内存使用情况?

但那些似乎允许您在调用 heap() 方法(在 guppy 的情况下)时更多地跟踪内存使用情况.但是，我想要跟踪的是外部库中的一个我无法修改的函数，它会增长以使用大量 RAM，但一旦函数执行完成就会释放它.有什么办法可以查出函数调用期间使用的RAM总量是多少?

but those seem to allow you more to track memory usage at the time the heap() method (in the case of guppy) is called. However, what I want to track is a function in an external library which I can't modify, and which grows to use a lot of RAM but then frees it once the execution of the function is complete. Is there any way to find out what the total amount of RAM used during the function call was?

推荐答案

这个问题似乎很有趣，它让我有理由研究 Guppy/Heapy，为此我感谢你.

This question seemed rather interesting and it gave me a reason to look into Guppy / Heapy, for that I thank you.

我尝试了大约 2 个小时让 Heapy 监视函数调用/进程，而不用零运气修改其源.

I tried for about 2 hours to get Heapy to do monitor a function call / process without modifying its source with zero luck.

我确实找到了使用内置 Python 库完成任务的方法<代码>资源.请注意，文档并未指明 RU_MAXRSS 值返回什么.另一位 SO 用户指出以 kB 为单位.运行 Mac OSX 7.3 并看着我的系统资源在下面的测试代码中攀升，我相信返回的值以 Bytes 为单位，而不是 kBytes.

I did find a way to accomplish your task using the built in Python library resource. Note that the documentation does not indicate what the RU_MAXRSS value returns. Another SO user noted that it was in kB. Running Mac OSX 7.3 and watching my system resources climb up during the test code below, I believe the returned values to be in Bytes, not kBytes.

关于我如何使用 resource 库监控库调用的 10000 英尺视图是在单独的(可监控)线程中启动该函数并在主线程中跟踪该进程的系统资源线.下面是您需要运行的两个文件来测试它.

A 10000ft view on how I used the resource library to monitor the library call was to launch the function in a separate (monitor-able) thread and track the system resources for that process in the main thread. Below I have the two files that you'd need to run to test it out.

图书馆资源监视器 -

Library Resource Monitor - whatever_you_want.py

import resource
import time

from stoppable_thread import StoppableThread


class MyLibrarySniffingClass(StoppableThread):
    def __init__(self, target_lib_call, arg1, arg2):
        super(MyLibrarySniffingClass, self).__init__()
        self.target_function = target_lib_call
        self.arg1 = arg1
        self.arg2 = arg2
        self.results = None

    def startup(self):
        # Overload the startup function
        print "Calling the Target Library Function..."

    def cleanup(self):
        # Overload the cleanup function
        print "Library Call Complete"

    def mainloop(self):
        # Start the library Call
        self.results = self.target_function(self.arg1, self.arg2)

        # Kill the thread when complete
        self.stop()

def SomeLongRunningLibraryCall(arg1, arg2):
    max_dict_entries = 2500
    delay_per_entry = .005

    some_large_dictionary = {}
    dict_entry_count = 0

    while(1):
        time.sleep(delay_per_entry)
        dict_entry_count += 1
        some_large_dictionary[dict_entry_count]=range(10000)

        if len(some_large_dictionary) > max_dict_entries:
            break

    print arg1 + " " +  arg2
    return "Good Bye World"

if __name__ == "__main__":
    # Lib Testing Code
    mythread = MyLibrarySniffingClass(SomeLongRunningLibraryCall, "Hello", "World")
    mythread.start()

    start_mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    delta_mem = 0
    max_memory = 0
    memory_usage_refresh = .005 # Seconds

    while(1):
        time.sleep(memory_usage_refresh)
        delta_mem = (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) - start_mem
        if delta_mem > max_memory:
            max_memory = delta_mem

        # Uncomment this line to see the memory usuage during run-time 
        # print "Memory Usage During Call: %d MB" % (delta_mem / 1000000.0)

        # Check to see if the library call is complete
        if mythread.isShutdown():
            print mythread.results
            break;

    print "
MAX Memory Usage in MB: " + str(round(max_memory / 1000.0, 3))

可停止线程 -

import threading
import time

class StoppableThread(threading.Thread):
    def __init__(self):
        super(StoppableThread, self).__init__()
        self.daemon = True
        self.__monitor = threading.Event()
        self.__monitor.set()
        self.__has_shutdown = False

    def run(self):
        '''Overloads the threading.Thread.run'''
        # Call the User's Startup functions
        self.startup()

        # Loop until the thread is stopped
        while self.isRunning():
            self.mainloop()

        # Clean up
        self.cleanup()

        # Flag to the outside world that the thread has exited
        # AND that the cleanup is complete
        self.__has_shutdown = True

    def stop(self):
        self.__monitor.clear()

    def isRunning(self):
        return self.__monitor.isSet()

    def isShutdown(self):
        return self.__has_shutdown


    ###############################
    ### User Defined Functions ####
    ###############################

    def mainloop(self):
        '''
        Expected to be overwritten in a subclass!!
        Note that Stoppable while(1) is handled in the built in "run".
        '''
        pass

    def startup(self):
        '''Expected to be overwritten in a subclass!!'''
        pass

    def cleanup(self):
        '''Expected to be overwritten in a subclass!!'''
        pass

这篇关于通过 Python 函数跟踪*最大*内存使用情况的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！