每行numpy的快速列随机播放 [英] Fast column shuffle of each row numpy

问题描述

我有一个长度超过10,000,000的大数组，其中包含行.我需要分别对这些行进行随机排序.例如:

I have a large 10,000,000+ length array that contains rows. I need to individually shuffle those rows. For example:

[[1,2,3]
 [1,2,3]
 [1,2,3]
 ...
 [1,2,3]]

到

[[3,1,2]
 [2,1,3]
 [1,3,2]
 ...
 [1,2,3]]

我当前正在使用

map(numpy.random.shuffle, array)

但这是一个python(不是NumPy)循环，占用了我99％的执行时间.可悲的是，PyPy JIT没有实现numpypy.random，所以我很不走运.有没有更快的方法?我愿意使用任何库(pandas，scikit-learn，scipy，theano等，只要它使用Numpy ndarray或派生类.)

But it's a python (not NumPy) loop and it's taking 99% of my execution time. Sadly, the PyPy JIT doesn't implement numpypy.random, so I'm out of luck. Is there any faster way? I'm willing to use any library (pandas, scikit-learn, scipy, theano, etc. as long as it uses a Numpy ndarray or a derivative.)

如果没有，我想我会使用Cython或C ++.

If not, I suppose I'll resort to Cython or C++.

推荐答案

以下是一些想法:

In [10]: a=np.zeros(shape=(1000,3))

In [12]: a[:,0]=1

In [13]: a[:,1]=2

In [14]: a[:,2]=3

In [17]: %timeit map(np.random.shuffle, a)
100 loops, best of 3: 4.65 ms per loop

In [21]: all_perm=np.array((list(itertools.permutations([0,1,2]))))

In [22]: b=all_perm[np.random.randint(0,6,size=1000)]

In [25]: %timeit (a.flatten()[(b+3*np.arange(1000)[...,np.newaxis]).flatten()]).reshape(a.shape)
1000 loops, best of 3: 393 us per loop

如果只有几列，那么所有可能的排列的数目都比数组中的行数小得多(在这种情况下，当只有3列时，只有6个可能的排列).使其更快的一种方法是先进行所有排列，然后通过从所有可能的排列中随机选择一个排列来重新排列每一行.

If there are only a few columns, then the number of all possible permutation is much smaller than the number of rows in the array (in this case, when there are only 3 columns, there are only 6 possible permutations). A way to make it faster is to make all the permutations at once first and then rearrange each row by randomly picking one permutation from all possible permutations.

即使尺寸较大，它仍然看起来要快10倍:

It still appears to be 10 times faster even with larger dimension:

#adjust a accordingly
In [32]: b=all_perm[np.random.randint(0,6,size=1000000)]

In [33]: %timeit (a.flatten()[(b+3*np.arange(1000000)[...,np.newaxis]).flatten()]).reshape(a.shape)
1 loops, best of 3: 348 ms per loop

In [34]: %timeit map(np.random.shuffle, a)
1 loops, best of 3: 4.64 s per loop

这篇关于每行numpy的快速列随机播放的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！