如何在现有的多处理池中增加特定数量的额外工作人员? [英] how to add specific number of additional workers to an exisiting multiprocessing pool?

问题描述

在以下情况下，我创建了具有两个工作程序的默认池并执行任务.在任务处理过程中，会定期检查task_queue，使其不会超过特定的长度限制，并防止上/下流混乱.如何动态添加更多的工作程序以减少任务队列长度?

In below situation I've created a default pool with two workers and perform tasks. During task processing the task_queue is checked regularly so it doesn't exceeds a certain length limit and prevents up/down stream clutter. How to add dynamically more workers to reduce the task queue length?

import multiprocessing as mp

... code snippet...

def main(poolsize, start_process):

    pool = mp.Pool(processes=poolsize, initializer=start_process)
    done = False

    task_queue = []

    while True:

        ... snippet code : do something ...

        if len(task_queue) >= 10:

            ... code to expand pool goes here...

        if done == True:
            break

    .. do final something ...

if __name__ == '__main__':

#    freeze_support()

    poolsize = 2

    main(poolsize)

推荐答案

要在运行池处理作业期间添加更多工作程序，可以在while循环内添加以下函数:

To add more workers during a running pool processing job you can add below function within the while-loop:

def repopulate(pool, add_workers):

    current_pool_size = len(pool._pool)         # _.pool gets the current pool size.

    new_pool_size = current_pool_size + add_workers

    pool._processes = new_pool_size

    pool._repopulate_pool()

    return pool

在main()的while循环内:

Within the while-loop from main():

if len(task_queue) >= 10:

    new_workers = 2

    repopulate(poolname, new_workers)

这篇关于如何在现有的多处理池中增加特定数量的额外工作人员?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！