如何在Python中并行化列表理解计算? [英] How to parallelize list-comprehension calculations in Python?
问题描述
至少在理论上,列表推导和映射计算都应该相对容易并行化:列表推导内的每个计算都可以独立于所有其他元素的计算来完成.例如在表达式中
Both list comprehensions and map-calculations should -- at least in theory -- be relatively easy to parallelize: each calculation inside a list-comprehension could be done independent of the calculation of all the other elements. For example in the expression
[ x*x for x in range(1000) ]
每个x * x计算都可以(至少在理论上)并行完成.
each x*x-Calculation could (at least in theory) be done in parallel.
我的问题是:是否有任何Python模块/Python实现/Python编程技巧来并行化列表理解计算(以便使用所有16/32/...内核或在计算机上分发计算-网格还是通过云)?
My question is: Is there any Python-Module / Python-Implementation / Python Programming-Trick to parallelize a list-comprehension calculation (in order to use all 16 / 32 / ... cores or distribute the calculation over a Computer-Grid or over a Cloud)?
推荐答案
As Ken said, it can't, but with 2.6's multiprocessing module, it's pretty easy to parallelize computations.
import multiprocessing
try:
cpus = multiprocessing.cpu_count()
except NotImplementedError:
cpus = 2 # arbitrary default
def square(n):
return n * n
pool = multiprocessing.Pool(processes=cpus)
print(pool.map(square, range(1000)))
文档中也有一些示例,展示了如何操作使用管理器,也应该允许分布式计算.
There are also examples in the documentation that show how to do this using Managers, which should allow for distributed computations as well.
这篇关于如何在Python中并行化列表理解计算?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!