如何解压来自"Pool.map()"的结果? [英] How to unpack results from `Pool.map()`?

问题描述

我有一个函数(preprocess_fit)，它将首先对数据集进行预处理(即平滑，基线校正和滤除不良数据).然后，该函数对参数进行初始猜测，然后在猜测中进行迭代以找到优化的拟合，然后返回const1，const2.该函数还计算了一堆其他参数，但是在这种情况下不会返回它们.

I've got a function (preprocess_fit) that will first preprocess a data set (i.e. smoothing, baseline correction, and filters out bad data). The function then takes an initial guess for a parameter and then iterates through guesses to find the optimised fit and then returns const1, const2. The function also calculates a bunch of other parameters but they are not returned in this instance.

然后，我需要在目录中的所有文件(〜1000个文件)上循环使用此功能.我通过使用包含for循环的第二个函数(函数)来实现此目的.预处理步骤(尤其是猜测迭代)特别耗时.

I then need to loop this function over all files in a directory (~1000 files). I do this by using the second function (function) that contains a for loop. The preprocessing step, in particular the guess iteration is particularly time consuming.

我想使用多处理模块来合并功能(函数)，并解压缩常量，然后附加到列表中.该try:except:被包括在内，因为某些文件缺少元数据并且preprocess_fit函数失败，并且我希望在发生这种情况时将nan值附加到列表中.

I'd like to pool the function (function) using the multiprocessing module and unpack the constants and then append to a list. The try: except: is included as some files are missing metadata and the preprocess_fit function fails and I'd like a nan value to be appended to the list when this occurs.

问题: 1)池无法打开功能 2)如果仅从函数(文件)返回const1，则进程将附加到列表中，而不是输出中.

Issues: 1) Pool cannot unpack function 2) If I only return a const1 from function(files) the processes are appended to the list and not the outputs.

任何建议都会很棒.

def preprocess_fit(file):
    #applies a number of pre-processing steps based on file metadata
    #optimizes fit starting with an initial guess for a parameter until RMS 
    #is minimized
    #returns constants from fitting process and final "guess" parameter
    return const1, const2

def function(files):
    for file in files:
        const1, const2 = preprocess_fit(file)
    return const1, const2

if __name__ == '__main__':
    files = glob.glob("largedata\*.txt")
    p = Pool(24)
    c1 = []
    c2 = []
    import numpy as np
    try:
        const1, const2 = p.map(function, files)
        c1.append(const1)
        c2.append(const2)
    except:
        c1.append(np.nan)
        c2.append(np.nan)
    p.close()
    p.join()

推荐答案

当函数返回多个项目时，您将从pool.map()调用中获得结果元组列表. const1将需要这些元组中的所有第一项，const2将需要这些元组中的所有第二项.这是 zip 内置函数的工作，该函数返回迭代器汇总了作为参数传递的每个可迭代对象中的元素.

When your function is returning multiple items, you will get a list of result-tuples from your pool.map() call. const1 would need all first items in these tuples, const2 all second items in these tuples. That's a job for the zip builtin-function, which returns an iterator that aggregates elements from each of the iterables passed as arguments.

您必须解压缩列表，以便结果元组是zip函数的参数.然后通过分配多个变量来解压缩迭代器:

You have to unpack the list so the result-tuples are the arguments for the zip function. Then unpack the iterator by assigning to multiple variables:

const1, const2 = zip(*pool.map(function, files))

这篇关于如何解压来自"Pool.map()"的结果?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！