大卫星图像处理 [英] Big Satellite Image Processing
问题描述
我正在尝试运行Mort Canty的 http://mcanty.homepage.t-online.de/二进制的RapidEye多光谱图像上的Python iMAD实现.它基本上计算了两个图像的规范相关性，然后将它们相减.我遇到的问题是 图片为5000 x 5000 x 5(带)像素.如果我尝试在 整个图像出现内存错误.
Im trying to run Mort Canty's http://mcanty.homepage.t-online.de/ Python iMAD implementation on bitemporal RapidEye Multispectral images. Which basically calculates the canonical correlation for the two images and then substracts them. The problem I'm having is that the images are of 5000 x 5000 x 5 (bands) pixels. If I try to run this on the whole image I get a memory error.
使用诸如pyTables之类的东西可以帮助我吗?
Would the use of something like pyTables help me with this?
Mort Canty的代码尝试执行的操作是使用gdal加载图像，然后将其存储 以10 x 25,000,000的阵列排列.
What Mort Canty's code tries to do is that it loads the images using gdal and then stores them in an 10 x 25,000,000 array.
# initial weights
wt = ones(cols*rows)
# data array (transposed so observations are columns)
dm = zeros((2*bands,cols*rows))
k = 0
for b in pos:
band1 = inDataset1.GetRasterBand(b+1)
band1 = band1.ReadAsArray(x0,y0,cols,rows).astype(float)
dm[k,:] = ravel(band1)
band2 = inDataset2.GetRasterBand(b+1)
band2 = band2.ReadAsArray(x0,y0,cols,rows).astype(float)
dm[bands+k,:] = ravel(band2)
k += 1
即使仅创建10 x 25,000,000 numpy的浮点数组也会引发内存错误.任何人都对如何解决这个问题有个好主意?这是我有史以来的第一篇文章，因此也欢迎任何关于如何发表的建议.
Even just creating a 10 x 25,000,000 numpy array of floats throws a memory error. Anyone have a good idea of how to get around this? This is my first post ever so any advice on how to post would also be welcome.
问候
推荐答案
numpy默认情况下使用float64，因此您的dm -array占用2GB内存(8 * 10 * 25000000)，其他数组每个大约200MB(〜8 * 5000 * 5000).
numpy uses float64 per default, so your dm-array takes up 2GB of memory (8*10*25000000), the other arrays probably about 200MB (~8*5000*5000) each.
astype(float)返回一个新数组，因此您也需要存储空间-可能甚至不需要，因为将数据复制到结果数组时会隐式转换类型.
astype(float) returns a new array, so you need memory for that as well - and is probably not even needed as the type is implicitly converted when copying the data to the result array.
何时释放for循环中使用的内存取决于垃圾回收.这不考虑GetRasterBand，ReadAsArray的内存开销.
when the memory used in the for-loop is freed depends on garbage collection. and this doesn't consider the memory overhead of GetRasterBand, ReadAsArray.
您确定输入数据使用64位浮点数吗?如果它使用32位浮点数，则可以通过在数组上指定dtype='f'来轻松实现一半的内存使用.
are your sure your input data uses 64-bit floats? if it uses 32-bit floats, you could easyliy half the memory usage by specifying dtype='f' on your arrays.
这篇关于大卫星图像处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
