Python itertools.combinations（）内存问题 [英] Python itertools.combinations() memory problems

问题描述

我正在处理大量的物品组合（来自《英雄联盟》），大约有7200万个，所有这些物品都被组合到一个函数中，该函数可以计算它们的益处。

I'm processing a huge number of combinations of items (from League of Legends), about 72 million, all of which are fed into a function that calculates how beneficial they are.

我们正在尝试找到可能的最佳组合。

We're trying to find the best possible combination.

忽略这样的事实：从算法上讲，可能会有更好的方法来做到这一点，任何人都可以告诉我为什么会出现内存错误？

Ignoring the fact that there might be better ways, algorithmically speaking, to do this, can anyone tell me why I'm getting a memory error?

allpossiblei = itertools.combinations(items.keys(),5)
maxc = 0
i = 0
for combo in allpossiblei:
    icombo = [items[name] for name in combo]
    res, tcost = calcStats(icombo, 0.658,100,100)
    if res > maxc :
        maxc = res
        print str(res) + " " + str(res/tcost)
        print combo
        print float(i)/79208745.0
    if i % 500000 == 0:
        print str(float(i)/79208745.0) + "\n \n"
        gc.collect()
    i = i + 1

除了使用局部变量进行算术运算外，calcStats不会执行任何操作。

calcStats doesn't do anything except arithmetic using local variables.

这会迅速消耗2gb +的内存并在大约5分钟内退出。我以为itertools应该提供不会消耗大量内存的生成器？我什至加入了该gc.collect（）语句，但它似乎不起作用。有想法吗？

This rapidly eats up 2gb+ of memory and quits in about 5 minutes. I thought itertools was supposed to provide a generator that wouldn't use up a ton of memory? I even threw in that gc.collect() statement but it doesn't seem to work. Any ideas?

推荐答案

组合通过使用提供的整个迭代器来创建池。没有办法解决。有关功能，请参阅文档中的伪代码： http://docs.python.org /library/itertools.html#itertools.combinations

Combinations creates a pool by consuming the whole iterator provided. There is no way around that. See the pseudocode in the docs for the function: http://docs.python.org/library/itertools.html#itertools.combinations

这篇关于Python itertools.combinations（）内存问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！