在Python中实现并行循环 [英] Implement Parallel for loops in Python

问题描述

我有一个看起来像这样的Python程序:

I have a Python program which looks like this:

total_error = []
for i in range(24):
    error = some_function_call(parameters1, parameters2)
    total_error += error

函数'some_function_call'需要很多时间，我找不到降低该函数时间复杂度的简便方法.有没有一种方法可以减少执行并行任务时的执行时间，以后再将它们加到total_error中. 我尝试使用pool和joblib，但均不能成功使用.

The function 'some_function_call' takes a lot of time and I can't find an easy way to reduce time complexity of the function. Is there a way to still reduce the execution time while performing parallel tasks and later adding them up in total_error. I tried using pool and joblib but could not successfully use either.

推荐答案

您还可以在Python 3中使用concurrent.futures，这是一个比multiprocessing更简单的接口. 请参见以了解有关差异的更多详细信息.

You can also use concurrent.futures in Python 3, which is a simpler interface than multiprocessing. See this for more details about differences.

from concurrent import futures

total_error = 0

with futures.ProcessPoolExecutor() as pool:
  for error in pool.map(some_function_call, parameters1, parameters2):
    total_error += error

在这种情况下，parameters1和parameters2应该是列表或可迭代的，其大小与您要运行该函数的次数相同(根据您的示例为24次).

In this case, parameters1 and parameters2 should be a list or iterable of the same size as the number of times you want to run the function (24 times as per your example).

如果paramters<1,2>不是可迭代/可映射的，但是您只想运行该函数24次，则可以提交该函数的作业所需的次数，然后再使用回调获取结果.

If paramters<1,2> are not iterables/mappable, but you just want to run the function 24 times, you can submit the jobs for the function for the required number of times, and later acquire the result using a callback.

class TotalError:
    def __init__(self):
        self.value = 0

    def __call__(self, r):
        self.value += r.result()

total_error = TotalError()
with futures.ProcessPoolExecutor() as pool:
  for i in range(24):
    future_result = pool.submit(some_function_call, parameters1, parameters2)
    future_result.add_done_callback(total_error)

print(total_error.value)

这篇关于在Python中实现并行循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！