具有多个参数和返回值的并行的两个函数 [英] Two functions in parallel with multiple arguments and return values

问题描述

我有两个独立的功能.它们中的每一个都需要相当长的时间来执行.

def function1(arg):do_some_stuff_here返回结果1def 函数 2(arg1, arg2, arg3):do_some_stuff_here返回结果2

我想并行启动它们，获取它们的结果(知道哪个是哪个)并在之后处理结果.据我所知，多处理比 Python 2.7 中的线程更有效(GIL 相关问题).然而，我有点迷茫，到底是使用进程、池还是队列更好，以及如何在我的用例中以正确的 Pythonic 方式实现它们.

感谢任何帮助;)

解决方案

首先，Process、Pool 和 Queue 都有不同的用例.

<块引用>

Process 用于通过创建 Process 对象来生成进程.

from multiprocessing import Process定义方法1():打印在方法1中"打印在方法1中"定义方法2():打印在方法2中"打印在方法2中"p1 = Process(target=method1) # 创建进程对象 p1p1.start() # 启动进程 p1p2 = 进程(目标 = 方法 2)p2.start()

<块引用>

Pool 用于并行执行多个函数输入值.

from multiprocessing import Pool定义方法1(x):打印 x打印 x**2返回 x**2p = 池(3)结果 = p.map(method1, [1,4,9])打印结果 # 打印 [1, 16, 81]

<块引用>

队列用于进程间通信.

from multiprocessing import Process, Queue定义方法1(x，l1):打印在方法1中"打印在方法1中"l1.put(x**2)返回 x定义方法2(x，l2):打印在方法2中"打印在方法2中"l2.put(x**3)返回 xl1 = 队列()p1 = Process(target=method1, args=(4, l1, ))l2 = 队列()p2 = Process(target=method2, args=(2, l2, ))p1.start()p2.start()print l1.get() # 打印 16打印 l2.get() # 打印 8

现在，对于您的情况，您可以使用 Process &队列(第三种方法)或者你可以操纵池方法来工作(下)

import itertools从多处理导入池导入系统定义方法1(x):打印 x打印 x**2返回 x**2定义方法2(x):打印 x打印 x**3返回 x**3def unzip_func(a, b):返回 a, b定义分发器(option_args):option, args = unzip_func(*option_args) # 解压选项和参数attr_name = "方法" + str(option)# 根据选项参数创建 attr_namevalue = getattr(sys.modules[__name__], attr_name)(args)# 调用名为 'attr_name' 的函数，参数为 args返回值option_list = [1,2] # 用于选择方法编号args_list = [4,2]# 对应方法的参数列表，(参数 4 用于方法 1)p = Pool(3) # 创建 3 个进程的池结果 = p.map(distributor, itertools.izip(option_list, args_list))# 调用分发器函数，参数被 itertools 包压缩为 (option1, arg1), (option2, arg2)打印结果 # 打印 [16,8]

希望这会有所帮助.

I've got two separate functions. Each of them takes quite a long time to execute.

def function1(arg):
     do_some_stuff_here
     return result1

def function2(arg1, arg2, arg3):
     do_some_stuff_here
     return result2

I'd like to launch them in parallel, get their results (knowing which is which) and process the results afterwards. For what I've understood, multiprocessing is more efficient than Threading in Python 2.7 (GIL related issue). However I'm a bit lost whether it is better to use Process, Pool or Queue and how to implement them in a correct pythonic way for my use case.

Any help appreciated ;)

解决方案

First of all, Process, Pool and Queue all have different use case.

Process is used to spawn a process by creating the Process object.

from multiprocessing import Process

def method1():
    print "in method1"
    print "in method1"

def method2():
    print "in method2"
    print "in method2"

p1 = Process(target=method1) # create a process object p1
p1.start()                   # starts the process p1
p2 = Process(target=method2)
p2.start()

Pool is used to parallelize execution of function across multiple input values.

from multiprocessing import Pool

def method1(x):
    print x
    print x**2
    return x**2

p = Pool(3)
result = p.map(method1, [1,4,9]) 
print result          # prints [1, 16, 81]

Queue is used to communicate between processes.

from multiprocessing import Process, Queue

def method1(x, l1):
    print "in method1"
    print "in method1"
    l1.put(x**2)
    return x

def method2(x, l2):
    print "in method2"
    print "in method2"
    l2.put(x**3)
    return x

l1 = Queue()
p1 = Process(target=method1, args=(4, l1, ))  
l2 = Queue()
p2 = Process(target=method2, args=(2, l2, )) 
p1.start()   
p2.start()      
print l1.get()          # prints 16
print l2.get()          # prints 8

Now, for your case you can use Process & Queue(3rd method) or you can manipulate the pool method to work (below)

import itertools
from multiprocessing import Pool
import sys

def method1(x):         
    print x
    print x**2
    return x**2

def method2(x):        
    print x
    print x**3
    return x**3

def unzip_func(a, b):  
    return a, b    

def distributor(option_args):
    option, args = unzip_func(*option_args)    # unzip option and args 

    attr_name = "method" + str(option)            
    # creating attr_name depending on option argument

    value = getattr(sys.modules[__name__], attr_name)(args) 
    # call the function with name 'attr_name' with argument args

    return value


option_list = [1,2]      # for selecting the method number
args_list = [4,2]        
# list of arg for the corresponding method, (argument 4 is for method1)

p = Pool(3)              # creating pool of 3 processes

result = p.map(distributor, itertools.izip(option_list, args_list)) 
# calling the distributor function with args zipped as (option1, arg1), (option2, arg2) by itertools package
print result             # prints [16,8]

Hope this helps.

这篇关于具有多个参数和返回值的并行的两个函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！