添加numpy数组时避免溢出 [英] Avoid overflow when adding numpy arrays

问题描述

我想用datatyp uint8添加numpy数组.我知道这些数组中的值可能足够大，以至于发生溢出.所以我得到类似的东西:

I want to add numpy arrays with datatyp uint8. I know that the values in these arrays may be large enough for an overflow to happen. So I get something like:

a = np.array([100, 200, 250], dtype=np.uint8)
b = np.array([50, 50, 50], dtype=np.uint8)
a += b

现在，a是[150 250 44].但是，我希望让uint8太大的值成为uint8允许的最大值，而不是溢出.所以我想要的结果是[150 250 255].

Now, a is [150 250 44]. However, instead of an overflow I want values which are too large for uint8 to be the maximum allowed for uint8. So my desired result would be [150 250 255].

我可以使用以下代码获得此结果:

I could get this result with the following code:

a = np.array([100, 200, 250], dtype=np.uint8)
b = np.array([50, 50, 50], dtype=np.uint8)
c = np.zeros((1,3), dtype=np.uint16)
c += a
c += b
c[c>255] = 255
a = np.array(c, dtype=np.uint8)

问题是我的数组很大，因此创建具有更大数据类型的第三个数组可能是内存问题.有没有一种快速且更高效的内存方式来实现上述结果?

The problem is, that my arrays are really big so creating a third array with a larger datatype could be a memory issue. Is there a fast and more memory efficient way to achieve the described result?

推荐答案

您可以通过创建第三个dtype uint8数组和一个bool数组(它们合起来比一个uint16数组更节省内存)来实现此目标/s>.

You can achieve this by creating a third array of dtype uint8, plus a bool array (which together are more memory efficient that one uint16 array).

np.putmask 是对于避免使用临时数组很有用.

np.putmask is useful for avoiding a temp array.

a = np.array([100, 200, 250], dtype=np.uint8)
b = np.array([50, 50, 50], dtype=np.uint8)
c = 255 - b  # a temp uint8 array here
np.putmask(a, c < a, c)  # a temp bool array here
a += b

---

但是，正如@moarningsun正确指出的那样，布尔数组占用的内存量与uint8数组相同，因此不一定有帮助.可以通过在任何给定时间避免有多个临时数组

a = np.array([100, 200, 250], dtype=np.uint8)
b = np.array([50, 50, 50], dtype=np.uint8)
b = 255 - b  # old b is gone shortly after new array is created
np.putmask(a, b < a, b)  # a temp bool array here, then it's gone
a += 255 - b  # a temp array here, then it's gone

这种方法将内存消耗换成CPU.

This approach trades memory consumption for CPU.

另一种方法是预先计算所有可能的结果，这是O(1)的额外内存(即与数组大小无关):

Another approach is to precalculate all possible results, which is O(1) extra memory (i.e. independent of the size of your arrays):

c = np.clip(np.arange(256) + np.arange(256)[..., np.newaxis], 0, 255).astype(np.uint8)
c
=> array([[  0,   1,   2, ..., 253, 254, 255],
          [  1,   2,   3, ..., 254, 255, 255],
          [  2,   3,   4, ..., 255, 255, 255],
          ..., 
          [253, 254, 255, ..., 255, 255, 255],
          [254, 255, 255, ..., 255, 255, 255],
          [255, 255, 255, ..., 255, 255, 255]], dtype=uint8)

c[a,b]
=> array([150, 250, 255], dtype=uint8)

如果您的阵列很大，这种方法是最有效的内存使用方法.再次，这是昂贵的处理时间，因为它用较慢的2dim数组索引代替了超快速整数加法.

This approach is the most memory-efficient if your arrays are very big. Again, it is expensive in processing time, because it replace the super-fast integer additions with the slower 2dim-array indexing.

如何工作

上面的c数组的构造使用了一个numpy广播技巧.将形状为(N,)的数组和形状为(1,N)的数组相加都广播为类似(N,N)的样子，因此结果是所有可能和的NxN数组.然后，我们剪辑它.我们得到一个满足以下条件的2dim数组:每个i，j c[i,j]=min(i+j,255).

Construction of the c array above makes use of a numpy broadcasting trick. Adding an array of shape (N,) and array of shape (1,N) broadcast both to be (N,N)-like, thus the result is an NxN array of all possible sums. Then, we clip it. We get a 2dim array that satisfies: c[i,j]=min(i+j,255) for each i,j.

然后剩下的就是使用花式索引索引来获取正确的值.使用您提供的输入，我们可以访问:

Then what's left is using fancy indexing the grab the right values. Working with the input you provided, we access:

c[( [100, 200, 250] , [50, 50, 50] )]

第一个索引数组引用第一个暗区，第二个索引数组指向第二个暗区.因此，结果是与索引数组((N,))具有相同形状的数组，由值[ c[100,50] , c[200,50] , c[250,50] ]组成.

The first index-array refers to the 1st dim, and the second to the 2nd dim. Thus the result is an array of the same shape as the index arrays ((N,)), consisting of the values [ c[100,50] , c[200,50] , c[250,50] ].

这篇关于添加numpy数组时避免溢出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！