我们如何在与Joblib并行执行中使用tqdm? [英] How can we use tqdm in a parallel execution with joblib?

问题描述

我想并行运行一个函数，并使用joblib等到所有并行节点都完成.像示例中一样:

I want to run a function in parallel, and wait until all parallel nodes are done, using joblib. Like in the example:

from math import sqrt
from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))

但是，我希望执行可以像 tqdm 一样在单个进度栏中看到，它显示了已经完成了多少工作.

But, I want that the execution will be seen in a single progressbar like with tqdm, showing how many jobs has been completed.

你会怎么做?

推荐答案

如果您的问题由很多部分组成，则可以将这些部分拆分为k个子组，并行运行每个子组，并在两个子组之间更新进度条，从而k更新进度.

If your problem consists of many parts, you could split the parts into k subgroups, run each subgroup in parallel and update the progressbar in between, resulting in k updates of the progress.

在文档的以下示例中对此进行了演示.

This is demonstrated in the following example from the documentation.

>>> with Parallel(n_jobs=2) as parallel:
...    accumulator = 0.
...    n_iter = 0
...    while accumulator < 1000:
...        results = parallel(delayed(sqrt)(accumulator + i ** 2)
...                           for i in range(5))
...        accumulator += sum(results)  # synchronization barrier
...        n_iter += 1

https://pythonhosted.org/joblib/parallel. html#reusing-a-pool-of-workers

这篇关于我们如何在与Joblib并行执行中使用tqdm?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！