创建集合的所有组合并耗尽内存 [英] Creating all combinations of a set and running out of memory

问题描述

我需要从55个集合中生成6个数字的每个组合.我相信该组合集合中有28,989,675个索引.我想我的内存不足了，因为我可以生成具有4个数字的组合，但没有比这大的数字了.我该如何解决这个问题?

I need to generate every combination of 6 numbers from a set of 55. I believe there are 28,989,675 indexes in that set of combinations. I guess I'm running out of memory, because I can generate combinations with 4 numbers but nothing larger than that. How can I fix this problem?

我正在使用从本教程中借来的一些代码的修改: https://www.youtube.com/watch?v=VyXDQxuIwPU

I'm using a modification of some code I borrowed from a tutorial here: https://www.youtube.com/watch?v=VyXDQxuIwPU

import itertools

text_file = open("comb3.txt", "w")

harmonics = [28, 33, 36, 38, 40, 43, 45, 47, 48, 50, 52, 55, 55.86, 57, 59, 60, 60.86, 61.69, 62, 63.86, 64, 65.86, 66, 66.69, 67, 69, 69.69, 70.86, 71, 71.69, 72, 74, 75.86, 76, 76.69, 77.86, 79, 81, 81.69, 82.86, 83.69, 84, 84.86, 86, 88, 88.69, 89.86, 90.69, 91, 93, 95, 95.69, 96.86, 98, 100]

combos = itertools.combinations(harmonics, 4)

usable_combos = []
for e in combos:
    usable_combos.append(e)

print usable_combos

s = str(usable_combos)

text_file.write(s)
text_file.close()

谢谢！

推荐答案

像itertools.combinations这样的迭代器一次只生成一段数据，这相对来说是内存有效的.但是，当您将所有值放入列表时，您需要内存来一次存储所有值(顺便说一句，usable_combos = list(combos)会替换您的for循环，而不是您应该这样做).

Iterators like itertools.combinations only generate a piece of data at a time which is relatively memory efficient. But when you put all of the values into a list you need memory to store all of them at once (btw, usable_combos = list(combos) would replace your for loop, not that you should do that).

由于将它们写入文件，因此可以在从迭代器获取文件时将每个组合写入文件，而无需创建列表.现在，您是否需要将其格式化为Python列表的repr?因为如果没有，那会更有意义:

Since you are writing them to a file, you can write each combo to the file as you get it from the iterator, not create a list. Now, do you need it to be formatted like the repr of a Python list? Because if not, this would make more sense:

for combo in combos:
    text_file.write(str(combo) + "\n")

注意:由于配置文件，从使用"{}\n".format(combo)更改.

Note: changed from using "{}\n".format(combo) due to profiling.

如果希望像列表的repr一样，则需要自己编写[和]，并用逗号代替换行符.

If you want it like the repr of the list, you'll need to write the [ and ] yourself, and commas instead of newlines.

-更多-

基于注释中的更新-如果您正在寻找特定的组合，寻找它们的最佳位置可能是在将它们写入文件之前，因为否则您只需要从文件中加载它们并查找再次对他们.如果您将根据某些条件从可用组合中选择一小部分，则提前选择它们会在以后减少您的工作.

Based on the updates in the comments - if you're looking for specific combinations, the best place to look for them is probably before writing them to the file, since otherwise you just have to load them back from the file and look at them all again. If you will be selecting a small fraction of the available combinations based on some criteria, selecting them up front will cut down your work later.

通常，您还可以在不学习实际C的情况下提高Cython的速度，并且如果您真的想暴力破解某些内存需求超出您自己计算机的东西，那么大小合适的EC2实例就在附近.每小时20美分.

In general, you could also look into Cython for some more speed without having to learn actual C, and if you really want to brute-force something with memory requirements beyond your own computer's, appropriately sized EC2 instances are in the vicinity of 20 cents an hour.

这篇关于创建集合的所有组合并耗尽内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！