初始化和销毁​​Python多处理工作者 [英] Initializing and destroying Python multiprocessing workers

问题描述

我有一个要从Python调用多次的模型.该模型需要很长时间才能启动和关闭，但是只需要很短的时间即可处理输入数据(可以在启动/关闭之间多次执行).多处理Pool()似乎是完成此任务的好方法，但是我很难让Model()类正确地销毁.

I have a model which I'm calling many times from Python. The model takes a long time to startup and shutdown, but only a short time to process the input data (which can be done multiple times between startup/shutdown). The multiprocessing Pool() seemed like a good way to get this done, but I'm having trouble getting the Model() class to destory correctly.

下面给出了程序代码的简化结构.实际上，init和del函数需要使用win32com.client模块做一些聪明的事情，而model.y变量是控制外部应用程序的句柄.

A simplified structure of the programs code is given below. In reality, the init and del functions need to do some clever things with the win32com.client module, and the model.y variable is a handle to control an external application.

#!/usr/bin/env python

import multiprocessing
import random
import time

class Model():
    def __init__(self):
        self.y = random.randint(0,5) # simplification
        time.sleep(5) # initialisation takes a while...
    def __del__(self):
        del(self.y) # simplification
        time.sleep(1) # destruction isn't especially quick either

def init():
    global model
    model = Model()

def f(x): # worker function, runs quickly
    return model.y * x ** 2

def main():
    pool = multiprocessing.Pool(processes=2, initializer=init)
    it = pool.imap(f, range(4))
    for n in range(4):
        print it.next()
    pool.close()

if __name__ == '__main__':
    main()

我从来没有想过要为Model()调用del函数，因为垃圾回收器中保留了一些引用.如何确保在程序结束时正确关闭模型?

The del function is never called for the Model()s, I'm guessing due to some reference being held in the garbage collector. How can I ensure that the model is closed correctly at the end of the program?

推荐答案

johnthexiii的解决方案将在第一次运行worker函数时终止该模型.您可以提供单独的销毁功能:

Solution of johnthexiii would kill the model at first run of worker function. You could offer a seperate destroy function:

import time

def g(x): # destroy function
    del model
    time.sleep(1) # to ensure, this worker does not pick too many
    return None

在添加pool.close()之前

pool.map_async(g, range(2), 1) # if specified to have two processes before

我不认为这是一个非常干净"的解决方案，但它应该可以工作.

I don’t think, this is a very "clean" solution, but it should work.

这篇关于初始化和销毁​​Python多处理工作者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！