itertools 产品加速 [英] itertools product speed up

问题描述

我使用 itertools.product 生成长度为 13 的 4 个元素的所有可能变化.4 和 13 可以是任意的，但实际上，我得到 4^13 个结果，这是很多.我需要将结果作为 Numpy 数组，目前执行以下操作:

 c = it.product([1,-1,np.complex(0,1), np.complex(0,-1)], repeat=length)sendbuf = np.array(list(c))

在中间插入一些简单的分析代码，看起来第一行几乎是即时的，而转换为列表然后转换为 Numpy 数组大约需要 3 个小时.有没有办法让它更快?这可能是我忽略的非常明显的事情.

谢谢！

解决方案

itertools.product() 的 NumPy 等价物是 numpy.indices()，但它会只为您提供 0,...,k-1 形式的范围的乘积:

numpy.rollaxis(numpy.indices((2, 3, 3)), 0, 4)数组([[[[0, 0, 0],[0, 0, 1],[0, 0, 2]],[[0, 1, 0],[0, 1, 1],[0, 1, 2]],[[0, 2, 0],[0, 2, 1],[0, 2, 2]]],[[[1, 0, 0],[1, 0, 1],[1, 0, 2]],[[1, 1, 0],[1, 1, 1],[1, 1, 2]],[[1, 2, 0],[1, 2, 1],[1, 2, 2]]]])

对于您的特殊情况，您可以使用

a = numpy.indices((4,)*13)b = 1j ** numpy.rollaxis(a, 0, 14)

(这不会在 32 位系统上运行，因为数组太大.从我可以测试的大小推断，它应该在不到一分钟的时间内运行.)

EIDT:顺便提一下:对 numpy.rollaxis() 的调用或多或少是装饰性的，以获得与 itertools.product() 相同的输出.如果你不关心索引的顺序，你可以省略它(但它很便宜，只要你没有任何将你的数组转换为连续数组的后续操作.)

获得

的精确模拟

numpy.array(list(itertools.product(some_list, repeat=some_length)))

你可以使用

numpy.array(some_list)[numpy.rollaxis(numpy.indices((len(some_list),) * some_length), 0, some_length + 1).reshape(-1, some_length)]

这完全不可读——告诉我是否应该进一步解释它:)

I use itertools.product to generate all possible variations of 4 elements of length 13. The 4 and 13 can be arbitrary, but as it is, I get 4^13 results, which is a lot. I need the result as a Numpy array and currently do the following:

  c = it.product([1,-1,np.complex(0,1), np.complex(0,-1)], repeat=length)
  sendbuf = np.array(list(c))

With some simple profiling code shoved in between, it looks like the first line is pretty much instantaneous, whereas the conversion to a list and then Numpy array takes about 3 hours. Is there a way to make this quicker? It's probably something really obvious that I am overlooking.

Thanks!

解决方案

The NumPy equivalent of itertools.product() is numpy.indices(), but it will only get you the product of ranges of the form 0,...,k-1:

numpy.rollaxis(numpy.indices((2, 3, 3)), 0, 4)
array([[[[0, 0, 0],
         [0, 0, 1],
         [0, 0, 2]],

        [[0, 1, 0],
         [0, 1, 1],
         [0, 1, 2]],

        [[0, 2, 0],
         [0, 2, 1],
         [0, 2, 2]]],


       [[[1, 0, 0],
         [1, 0, 1],
         [1, 0, 2]],

        [[1, 1, 0],
         [1, 1, 1],
         [1, 1, 2]],

        [[1, 2, 0],
         [1, 2, 1],
         [1, 2, 2]]]])

For your special case, you can use

a = numpy.indices((4,)*13)
b = 1j ** numpy.rollaxis(a, 0, 14)

(This won't run on a 32 bit system, because the array is to large. Extrapolating from the size I can test, it should run in less than a minute though.)

EIDT: Just to mention it: the call to numpy.rollaxis() is more or less cosmetical, to get the same output as itertools.product(). If you don't care about the order of the indices, you can just omit it (but it is cheap anyway as long as you don't have any follow-up operations that would transform your array into a contiguous array.)

EDIT2: To get the exact analogue of

numpy.array(list(itertools.product(some_list, repeat=some_length)))

you can use

numpy.array(some_list)[numpy.rollaxis(
    numpy.indices((len(some_list),) * some_length), 0, some_length + 1)
    .reshape(-1, some_length)]

This got completely unreadable -- just tell me whether I should explain it any further :)

这篇关于itertools 产品加速的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！