使用Python以真实顺序打印Parellel函数输出 [英] Printing a Parellel Function Outputs in True Order w/Python

问题描述

希望为Python并行化脚本按顺序打印所有内容.注意c3在b2之前打印-乱序.有什么办法让下面的功能具有等待功能?如果重新运行，有时对于较短的批次，打印顺序是正确的.但是，正在寻找可重现的解决方案.

from joblib import Parallel, delayed, parallel_backend
import multiprocessing

testFrame = [['a',1], ['b', 2], ['c', 3]]

def testPrint(letr, numbr):
  print(letr + str(numbr))
  return letr + str(numbr)

with parallel_backend('multiprocessing'):
  num_cores = multiprocessing.cpu_count()
  results = Parallel(n_jobs = num_cores)(delayed(testPrint)(letr = testFrame[i][0], 
        numbr = testFrame[i][1]) for i in range(len(testFrame))) 

print('##########')
for test in results:
  print(test)

输出:

b2
c3
a1
##########
a1
b2
c3

Seeking:
a1
b2
c3
##########
a1
b2
c3

解决方案

Once you launch tasks in separate processes you no longer control the order of execution so you cannot expect the actions of those tasks to execute in any predictable order - especially if the tasks can take varying lengths of time.

If you are parallelizing(?) a task/function with a sequence of arguments and you want to reorder the results to match the order of the original sequence you can pass sequence information to the task/function that will be returned by the task and can be used to reconstruct the original order.

If the original function looks like this:

def f(arg):
    l,n = arg
    #do stuff
    time.sleep(random.uniform(.1,10.))
    result = f'{l}{n}'
    return result

Refactor the function to accept the sequence information and pass it through with the return value.

def f(arg):
    indx, (l,n) = arg
    time.sleep(random.uniform(.1,10.))
    result = (indx,f'{l}{n}')
    return result

enumerate could be used to add the sequence information to the sequence of data:

originaldata = list(zip('abcdefghijklmnopqrstuvwxyz', range(26)))
dataplus = enumerate(originaldata)

Now the arguments have the form (index,originalarg) ... (0, ('a',0'), (1, ('b',1)).

And the returned values from the multi-processes look like this (if collected in a list) -

[(14, 'o14'), (23, 'x23'), (1, 'b1'), (4, 'e4'), (13, 'n13'),...]

Which is easily sorted on the first item of each result, key=lambda item: item[0], and the values you really want obtained by picking out the second items after sorting results = [item[1] for item in results].

这篇关于使用Python以真实顺序打印Parellel函数输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！