joblib 的中间结果 [英] Intermediate results from joblib

问题描述

我正在尝试学习 joblib 模块作为替代 Python 中内置的 multiprocessing 模块.我习惯于使用 multiprocessing.imap 在可迭代对象上运行一个函数并在结果进来时返回结果.在这个最小的工作示例中，我不知道如何使用 joblib:

I'm trying to learn the joblib module as an alternative to the builtin multiprocessing module in python. I'm used to using multiprocessing.imap to run a function over an iterable and returning the results as they come in. In this minimal working example, I can't figure out how to do it with joblib:

import joblib, time

def hello(n):
    time.sleep(1)
    print "Inside function", n
    return n

with joblib.Parallel(n_jobs=1) as MP:

    func = joblib.delayed(hello)
    for x in MP(func(x) for x in range(3)):
        print "Outside function", x

打印:

Inside function 0
Inside function 1
Inside function 2
Outside function 0
Outside function 1
Outside function 2

我想看看输出:

Inside function 0
Outside function 0
Inside function 1
Outside function 1
Inside function 2
Outside function 2

或类似的东西，表明可迭代的 MP(...) 不是在等待所有结果完成.对于更长的演示更改 n_jobs=-1 和 range(100).

Or something similar, indicating that the iterable MP(...) is not waiting for all the results to complete. For longer demo change n_jobs=-1 and range(100).

推荐答案

要从 joblib 中获得即时结果，例如:

from joblib._parallel_backends import MultiprocessingBackend

class ImmediateResult_Backend(MultiprocessingBackend):
    def callback(self, result):
        print("\tImmediateResult function %s" % (result))

    # Overload apply_async and set callback=self.callback
    def apply_async(self, func, callback=None):
        applyResult = super().apply_async(func, self.callback)
        return applyResult

joblib.register_parallel_backend('custom', ImmediateResult_Backend, make_default=True)

with joblib.Parallel(n_jobs=2) as parallel:
    func = parallel(delayed(hello)(y) for y in range(3))
    for f in func:
        print("Outside function %s" % (f))

输出:
注意:我在 def hello(...) 中使用了 time.sleep(n * random.randrange(1,5))，因此 processes> 准备好与众不同.

Output:
Note: I use time.sleep(n * random.randrange(1,5)) in def hello(...), therefore processes become different ready.

内部函数0
内部函数1
立即结果函数 [0]
内部函数2
立即结果函数 [1]
立即结果函数 [2]
外部函数 0
外功能1
外函数2

Inside function 0
Inside function 1
ImmediateResult function [0]
Inside function 2
ImmediateResult function [1]
ImmediateResult function [2]
Outside function 0
Outside function 1
Outside function 2

使用 Python:3.4.2 测试 - joblib:0.11

这篇关于joblib 的中间结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！