MemoryError:无法为形状为(323313，3435)和数据类型float64的数组分配8.27 GiB [英] MemoryError: Unable to allocate 8.27 GiB for an array with shape (323313, 3435) and data type float64

问题描述

我的数据框中有扩展名(例如.exe，.py，.xml，.doc等)表.在终端上运行后我在大型数据集上遇到了错误.

I have extension(example .exe,.py,.xml,.doc etc) table in my dataframe. after running on terminal I am getting above error on large data set.

encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit(features['Extension'].values.reshape(-1, 1))
temp = encoder.transform(features['Extension'].values.reshape(-1, 1)).toarray()  #GETTING ERROR on this
print("Size of array in bytes",getsizeof(temp))
print("Array :-",temp)
print("Shape :- ",features.shape, temp.shape)
features.drop(columns=['Extension'], axis=1, inplace=True)
dump(encoder, os.path.join(os.getcwd(), 'model_dumps', 'encoder.pkl'))
features.drop(columns=['Extension'], axis=1, inplace=True)
features = featureScaling(features)
features = np.concatenate((features, temp), axis=1)

输出-

1) Size of array in bytes :- 8884558912
2) Array :- 
[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [1. 0. 0. ... 0. 0. 0.]
 [1. 0. 0. ... 0. 0. 0.]
 [1. 0. 0. ... 0. 0. 0.]]
3)Shape :- (323310, 8) (323310, 3435)

推荐答案

这很有趣.

MemoryError:无法为形状(323313，3435)和数据类型float64的数组分配8.27 GiB

MemoryError: Unable to allocate 8.27 GiB for an array with shape (323313, 3435) and data type float64

大多数现代计算机的RAM都不超过8 Gb.看来您有8位使用者，而python无法将所有这些资料放入记忆体中.尝试购买另一台具有更多内存的计算机，或者升级现有的计算机.这肯定会解决问题.

most modern computers don't have more than 8 Gb of RAM. Looks like you have 8 and python is not able to fit all this data in the memory. Try buying another computer with more ram or upgrade your existing one. This will definitely fix the issue.

这篇关于MemoryError:无法为形状为(323313，3435)和数据类型float64的数组分配8.27 GiB的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！