使用Python以真实顺序打印Parellel函数输出 [英] Printing a Parellel Function Outputs in True Order w/Python
问题描述
希望为Python并行化脚本按顺序打印所有内容.注意c3在b2之前打印-乱序.有什么办法让下面的功能具有等待功能?如果重新运行,有时对于较短的批次,打印顺序是正确的.但是,正在寻找可重现的解决方案.
from joblib import Parallel, delayed, parallel_backend
import multiprocessing
testFrame = [['a',1], ['b', 2], ['c', 3]]
def testPrint(letr, numbr):
print(letr + str(numbr))
return letr + str(numbr)
with parallel_backend('multiprocessing'):
num_cores = multiprocessing.cpu_count()
results = Parallel(n_jobs = num_cores)(delayed(testPrint)(letr = testFrame[i][0],
numbr = testFrame[i][1]) for i in range(len(testFrame)))
print('##########')
for test in results:
print(test)
输出:
b2
c3
a1
##########
a1
b2
c3
Seeking:
a1
b2
c3
##########
a1
b2
c3
Once you launch tasks in separate processes you no longer control the order of execution so you cannot expect the actions of those tasks to execute in any predictable order - especially if the tasks can take varying lengths of time.
If you are parallelizing(?) a task/function with a sequence of arguments and you want to reorder the results to match the order of the original sequence you can pass sequence information to the task/function that will be returned by the task and can be used to reconstruct the original order.
If the original function looks like this:
def f(arg):
l,n = arg
#do stuff
time.sleep(random.uniform(.1,10.))
result = f'{l}{n}'
return result
Refactor the function to accept the sequence information and pass it through with the return value.
def f(arg):
indx, (l,n) = arg
time.sleep(random.uniform(.1,10.))
result = (indx,f'{l}{n}')
return result
enumerate
could be used to add the sequence information to the sequence of data:
originaldata = list(zip('abcdefghijklmnopqrstuvwxyz', range(26)))
dataplus = enumerate(originaldata)
Now the arguments have the form (index,originalarg)
... (0, ('a',0'), (1, ('b',1))
.
And the returned values from the multi-processes look like this (if collected in a list) -
[(14, 'o14'), (23, 'x23'), (1, 'b1'), (4, 'e4'), (13, 'n13'),...]
Which is easily sorted on the first item of each result, key=lambda item: item[0]
, and the values you really want obtained by picking out the second items after sorting results = [item[1] for item in results]
.
这篇关于使用Python以真实顺序打印Parellel函数输出的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!