如何在python中并行化big for循环 [英] how to parallelize big for loops in python
问题描述
我刚接触Python，但我仍处于学习曲线的艰难阶段.感谢您的任何评论.
I just got to Python, and I am still in the steep phase of the learning curve. Thank you for any comments ahead.
我有一个很大的for循环要运行(在许多迭代中都很大)，例如:
I have a big for loop to run (big in the sense of many iterations), for example:
for i in range(10000)
for j in range(10000)
f((i,j))
尽管我会问如何将其并行化，这是一个常见的问题，但在Google上搜索了数小时后，我使用多重处理"模块找到了解决方案，如下所示:
I though that it would be a common question of how to parallelize it, and after hours of search on google I arrived at the solution using "multiprocessing" module, as the following:
pool=Pool()
x=pool.map(f,[(i,j) for i in range(10000) for j in range(10000)])
这在循环较小时起作用.但是，如果循环很大，那确实很慢；或者如果循环太大，有时会发生内存错误.看来python会首先生成参数列表，然后甚至使用xrange将列表提供给函数"f".那是对的吗?
This works when the loop is small. However, it is really slow if the loop is large, Or sometimes a memory error occurs if the loops are too big. It seems that python would generate the list of arguments first, and then feed the list to the function "f", even using xrange. Is that correct?
所以这种并行化对我来说不起作用，因为我真的不需要将所有参数都存储在列表中.有一个更好的方法吗?我感谢任何建议或参考.谢谢.
So this parallelization does not work for me because I do not really need to store all arguments in a list. Is there a better way to do this? I appreciate any suggestions or references. Thank you.
推荐答案
似乎python会首先生成参数列表，然后甚至使用xrange将列表提供给函数"f".正确吗?
It seems that python would generate the list of arguments first, and then feed the list to the function "f", even using xrange. Is that correct?
是的，因为您使用的是列表推导，它会明确要求它生成该列表.
Yes, because you're using a list comprehension, which explicitly asks it to generate that list.
(请注意，这里的xrange并不重要，因为一次只能有两个范围，每个范围10K；与参数列表的100M相比，没什么.)
(Note that xrange isn't really relevant here, because you only have two ranges at a time, each 10K long; compared to the 100M of the argument list, that's nothing.)
如果希望它根据需要动态生成值，而不是一次生成所有100M，则要使用生成器表达式而不是列表推导.几乎总是将括号变成括号即可.
If you want it to generate the values on the fly as needed, instead of all 100M at once, you want to use a generator expression instead of a list comprehension. Which is almost always just a matter of turning the brackets into parentheses:
x=pool.map(f,((i,j) for i in range(10000) for j in range(10000)))
但是，正如您从源，如果您给map提供一个生成器，它最终只会列出一个列表，因此在这种情况下，它什么也解决不了. (文档没有明确说明这一点，但是很难看到如何选择合适的块大小将可迭代对象切成小段长度……).
However, as you can see from the source, map will ultimately just make a list if you give it a generator, so in this case, that won't solve anything. (The docs don't explicitly say this, but it's hard to see how it could pick a good chunksize to chop the iterable into if it didn't have a length…).
而且，即使事实并非如此，您仍然会在结果上再次遇到相同的问题，因为pool.map返回一个列表.
And, even if that weren't true, you'd still just run into the same problem again with the results, because pool.map returns a list.
要解决这两个问题，可以使用
To solve both problems, you can use pool.imap instead. It consumes the iterable lazily, and returns a lazy iterator of results.
要注意的一件事是，如果不传递imap，它不会猜测最佳的块大小，而只是默认为1，因此您可能需要一些思考或尝试和错误来对其进行优化.
One thing to note is that imap does not guess at the best chunksize if you don't pass one, but just defaults to 1, so you may need a bit of thought or trial&error to optimize it.
此外，imap仍会在输入结果时将它们排队，因此可以按照与参数相同的顺序将它们反馈给您.在病理情况下，它可能最终导致结果排队(poolsize-1)/poolsize，尽管在实践中这种情况很少见.如果要解决此问题，请使用imap_unordered.如果您需要了解顺序，只需将参数与参数和结果来回传递即可:
Also, imap will still queue up some results as they come in, so it can feed them back to you in the same order as the arguments. In pathological cases, it could end up queuing up (poolsize-1)/poolsize of your results, although in practice this is incredibly rare. If you want to solve this, use imap_unordered. If you need to know the ordering, just pass the indexes back and forth with the args and results:
args = ((i, j) for i in range(10000) for j in range(10000))
def indexed_f(index, (i, j)):
return index, f(i, j)
results = pool.imap_unordered(indexed_f, enumerate(args))
但是，我注意到在您的原始代码中，您对f(i, j)的结果根本不做任何事情.在那种情况下，为什么还要费心收集所有结果呢?在这种情况下，您可以返回循环:
However, I notice that in your original code, you're not doing anything at all with the results of f(i, j). In that case, why even bother gathering up the results at all? In that case, you can just go back to the loop:
for i in range(10000):
for j in range(10000):
map.apply_async(f, (i,j))
但是，imap_unordered仍然值得使用，因为它提供了一种非常简单的方法来阻止所有任务完成，同时仍使池本身处于运行状态以供以后使用:
However, imap_unordered may still be worth using, because it provides a very easy way to block until all of the tasks are done, while still leaving the pool itself running for later use:
def consume(iterator):
deque(iterator, max_len=0)
x=pool.imap_unordered(f,((i,j) for i in range(10000) for j in range(10000)))
consume(x)
这篇关于如何在python中并行化big for循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
