Python:将数据类型从整数更改为浮点数时出现内存错误 [英] Python: memory error while changing data type from integer to float

问题描述

我有一个大小为 13000*300000 的数组，其中填充了从 0 到 255 的整数.我想将它们的数据类型从整数更改为浮点数，就像数据是一个 numpy 数组一样:

I have a array of size 13000*300000 filled with integer from 0 to 255. I would like to change their data type from integer to float as if data is a numpy array:

 data.astype('float')

将其数据类型从整数更改为浮点数时，显示内存错误.我有 80 GB 的内存.它仍然显示内存错误.你能告诉我可能是什么原因吗?

While changing its data type from integer to float, it shows memory error. I have 80 GB of RAM. It still shows memory error. Could you please let me know what can be the reason for it?

推荐答案

这里的问题是数据量很大(大约 30GB sequential 数据，见 numpy 数组中有多少内存?)，因此它在尝试时导致错误将其放入内存中.与其做整体操作，不如先切片再做操作再合并，比如:

The problem here is that data is huge (about 30GB of sequential data, see How much memory in numpy array?), hence it causes the error while trying to fit it into the memory. Instead of doing the operation on whole, slice it and then do the operation and then merge, like:

n = 300000
d1 = data[:, :n/2].astype('float')
d2 = data[:, n/2:].astype('float')

data = np.hstack(d1, d2)

通常，由于您的数据量太大，请考虑将其分部分使用，以免一直被此类问题困扰(请参阅 处理大型 Numpy 数组的技术? 用于此技术和其他技术).

Generally, since your data size is so unwieldy, consider consuming it in parts to avoid being bitten by these sorts of problems all the time (see Techniques for working with large Numpy arrays? for this and other techniques).

这篇关于Python:将数据类型从整数更改为浮点数时出现内存错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！