如何实现对python multiprocessing.Pool的自定义控件? [英] How to implement custom control over python multiprocessing.Pool?

问题描述

通常，我使用以下代码，并且无论您以何种顺序使用函数process_func来处理某些参数，它都可以正常工作:

Usually i use following code, and it works fine when you do not matter in which order function process_func will handle some parameter:

params = [1,2,3,4,5 ... ]

def process_func():
    ...

pool = new Pool(40)
pool.map(process_func, params)
pool.close()
pool.join()

在上面的示例中，我们有一种类型的进程，最大同时数量为40.但是，假设我们有不同类型的进程(参数)，应同时执行这些进程(参数).例如，在我的硒网格中，我有40个Firefox，40个铬.我有5000个测试用例，其中一些更喜欢chrome，其中一些更喜欢Firefox，其中一些并不重要.

In example above we have processes of one type, with maximum simultanious number of 40. But.. imagine we have processes (parameters) of different type, which should be executed simultaniously. For example, in my selenium grid i have 40 firefox, 40 chromes. And i have 5000 test cases, some of them prefer chrome, some of them firefox, some of them does not matter.

例如，假设我们具有以下类型:

For example, lets say we have following types:

• firefox类型:最大同时编号:40
• chrome类型:最大同时编号:40

在这种情况下，我们的池最多可以有80个同步过程，但是有一个严格的规则:其中40个必须是Firefox，其中40个必须是Chrome.

In this case our pool will have maximum of 80 simultanious processes, but there is strict rule: 40 of them must be firefox, 40 of them must be chromes.

这意味着不会互相接连使用参数.池必须以某种方式从参数列表中选择值，以使每种处理类型都具有最大值.

It means that params won't be taken one after each other. Pool must select value from params list in a way to have maximum of each process type.

如何实现?

推荐答案

我将修改您的process_func()以使用另一个参数，告诉它是哪个类型"，并使用两个单独的池.添加 functools.partial 将使我们仍然使用pool.map() :

I would modify your process_func() to take one more parameter that tells it which "type" to be and use two separate pools. Adding functools.partial will allow us to still use pool.map():

from functools import partial
from multiprocessing import Pool

params = [1,2,3,4,5 ... ]

def process_func(type, param):
    if type == 'Firefox':
        # do Firefox stuff
    else:
        # do Chrome stuff

chrome_pool = Pool(40)
fox_pool = Pool(40)

chrome_function = partial(process_func, 'Chrome')
fox_function = partial(process_func, 'Firefox')

chrome_pool.map(chrome_func, params)
fox_pool.map(fox_func, params)

chrome_pool.close()
fox_pool.close()
chrome_pool.join()
fox_pool.join()

functools.partial()函数允许我们通过返回始终提供该参数的新函数对象，将参数绑定到特定值.这种方法允许您将每个类型"(由于缺乏更好的用语)限制为40个工作进程.

The functools.partial() function allows us to bind an argument to a specific value, by returning a new function object that will always supply that argument. This approach allows you to limit each "type" (for lack of a better term) to 40 worker processes.

这篇关于如何实现对python multiprocessing.Pool的自定义控件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！