如何在 Python 中并行化列表理解计算? [英] How to parallelize list-comprehension calculations in Python?

问题描述

列表理解和映射计算都应该——至少在理论上——相对容易并行化:列表理解中的每个计算都可以独立于所有其他元素的计算来完成.例如在表达式中

Both list comprehensions and map-calculations should -- at least in theory -- be relatively easy to parallelize: each calculation inside a list-comprehension could be done independent of the calculation of all the other elements. For example in the expression

[ x*x for x in range(1000) ]

每个 x*x 计算(至少在理论上)可以并行完成.

each x*x-Calculation could (at least in theory) be done in parallel.

我的问题是:是否有任何 Python-Module/Python-Implementation/Python Programming-Trick 来并行化列表理解计算(为了使用所有 16/32/... 内核或将计算分布在计算机上-网格或云)?

My question is: Is there any Python-Module / Python-Implementation / Python Programming-Trick to parallelize a list-comprehension calculation (in order to use all 16 / 32 / ... cores or distribute the calculation over a Computer-Grid or over a Cloud)?

推荐答案

正如 Ken 所说，它不能，但使用 2.6 的 multiprocessing 模块，很容易并行计算.

As Ken said, it can't, but with 2.6's multiprocessing module, it's pretty easy to parallelize computations.

import multiprocessing

try:
    cpus = multiprocessing.cpu_count()
except NotImplementedError:
    cpus = 2   # arbitrary default


def square(n):
    return n * n

pool = multiprocessing.Pool(processes=cpus)
print(pool.map(square, range(1000)))

文档中还有一些示例说明了如何操作这使用管理器，它也应该允许分布式计算.

There are also examples in the documentation that show how to do this using Managers, which should allow for distributed computations as well.

这篇关于如何在 Python 中并行化列表理解计算?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！