为什么我的itertools.product出现MemoryError? [英] Why do I get a MemoryError with itertools.product?

问题描述

我希望下面的代码片段能给我一个迭代器，该迭代器从两个输入可迭代对象的笛卡尔积中产生对:

I would expect the following snippet to give me an iterator yielding pairs from the Cartesian product of the two input iterables:

$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import itertools
>>> one = xrange(0, 10**9)
>>> two = (1,)
>>> prods = itertools.product(one, two)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

相反，我得到一个MemoryError.但是我认为 itertools.product 不会将中间结果存储在内存中，那么是什么原因导致MemoryError?

Instead, I get a MemoryError. But I thought that itertools.product did not store the intermediate results in memory, so what's causing the MemoryError?

推荐答案

它不存储中间的结果，但是它必须存储输入值，因为每个输入值可能需要多次使用几个输出值.

It doesn't store intermediate results, but it has to store the input values because each of those might be needed several times for several output values.

由于只能在迭代器上迭代一次，因此product不能等效于此实现:

Since you can only iterate once over an iterator, product cannot be implemented equivalent to this:

def prod(a, b):
  for x in a:
    for y in b:
       yield (x, y)

如果此处b是迭代器，则它将在外循环的第一次迭代后耗尽，并且在随后的for y in b执行中将不再产生任何元素.

If here b is an iterator, it will be exhausted after the first iteration of the outer loop and no more elements will be produced in subsequent executions of for y in b.

product通过存储b产生的所有元素来解决此问题，以便可以重复使用:

product works around this problem by storing all the elements that are produced by b, so that they can be used repeatedly:

def prod(a, b):
  b_ = tuple(b)  # create tuple with all the elements produced by b
  for x in a:
    for y in b_:
       yield (x, y)

实际上，product尝试存储由给定的所有可迭代对象生成的元素，即使可以避免使用其第一个参数.该函数只需要遍历第一个可迭代对象一次，因此不必缓存这些值.但无论如何，它都会尝试执行操作，从而导致您看到MemoryError.

In fact, product tries to store the elements produced by all the iterables it is given, even though that could be avoided for its first parameter. The function only needs to walk over the first iterable once, so it wouldn't have to cache those values. But it tries to do anyway, which leads to the MemoryError you see.

这篇关于为什么我的itertools.product出现MemoryError?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！