函数内部多处理科学优化的怪异行为 [英] weird behavior of multiprocessing scipy optimization inside of a function

问题描述

这是一个运行良好的简单代码.即使函数最小化包装了scipy.optimize.minimize它也不会抱怨腌制

Here is a simple code that runs well. Even if the function minimize wraps the scipy.optimize.minimize it does not complain about pickling

import numpy as np
from scipy import optimize
from multiprocessing import Pool

def square(x):
    return np.sum(x**2+ 2*x)

def minimize(args):
    f,x = args
    res = optimize.minimize(f, x, method = 'L-BFGS-B')
    return res.x

x = np.random.rand(8,10)

args = [(square,x[i]) for i in range(8)]
p = Pool(8)
p.map(minimize,args)

但是，如果尝试以下操作，则会由于酸洗错误而失败

However, if try the following it fails with the pickling error

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

def run():
    def square(x):
        return np.sum(x**2+ 2*x)

    def minimize(args):
        f,x = args
        res = optimize.minimize(f, x, method = 'L-BFGS-B')
        return res.x

    x = np.random.rand(8,10)

    args = [(square,x[i]) for i in range(8)]
    p = Pool(8)
    p.map(minimize,args)

run()

我想制作一个模块，以便在许多初始猜测的同时使用scipy最小化.但是，如示例中所示，当我将其设为模块时，它将失败.

I want to make a module to use scipy minimize in parallel with many population of initial guesses. However, as shown in the example, when I make it a module, it fails.

推荐答案

问题是Python无法腌制嵌套函数，在第二个示例中，您试图将嵌套的minimize和square函数传递给您的孩子的过程，这需要酸洗.

The problem is that Python cannot pickle nested functions, and in your second example, you're trying to pass the nested minimize and square functions to your child process, which requires pickling.

如果没有理由必须嵌套这两个函数，请将它们移到模块的顶层将解决此问题.您还可以参阅此问题，以了解腌制嵌套函数的技术.

If there's no reason that you must nest those two functions, moving them to the top-level of the module will fix the issue. You can also see this question for techniques to pickle nested functions.

这篇关于函数内部多处理科学优化的怪异行为的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！