Java中的大数组导致的内存不足异常 [英] out of memory exception caused by big array in java

问题描述

我正在实现一种基于概率潜在语义索引(plsa)的算法，本文为这里，它需要一个四维数组，名称为p_z_d_wt_wv，z是主题，d是文档，wt是文本词，wv是可视词，每个维数约为12， 7000,100，500，并且该数组是双精度数组，因此需要 32G内存！ 我按照下面的方式分配此内存，这只是为了演示，因为每个文档中wt和wv的数量是不同的.

I'm implenmenting an algorithm which based on probabilistic latent semantic indexing(plsa) and the paper is hereand it need a four dimension array which named p_z_d_wt_wv, z is topic, d is document, wt is text word, wv is visual word,and the number of each dimension is about 12, 7000,100, 500, and the array is a double array, so it need 32G memory!! I allocate this memory like this way below, and it is just for demonstration as the number of wt and wv in each document is different.

p_z_d_wt_wv = new double[12][7000][][]; 
for( int t = 0; t < 12; ++t) 
{ 
    for( int d = 0; d < 7000; ++d ) 
    { 
        p_z_d_wt_wv[t][d] = new double[100][500];
    } 
}

当我运行代码时，它有内存不足的问题. 首先，为什么我的代码用完了内存? 如果以我的方式分配数组，是否连续分配内存?是因为java对连续内存有内存限制吗?如果是这样，有什么限制?

when I run the code, it has out of memory problem. First, why do my code run out of memory? Are the memory allocated consecutively if the array are allocated in my way? Is it because java have a memory limit for consecutive memory? If so, what's the limit?

第二，假设服务器的内存足够大，我该怎么办才能解决此问题. 我知道我可以将其更改为 float 数组，但是还有其他解决方案吗?

Second, what can I do to solve this problem supposed that the memory of the server is big enough. I know I can change it as a float array, but are there any other solutions?

推荐答案

如果您确实需要所有内存，那么，您就需要所有内存.

If you actually need all of that memory, well, you need all of that memory.

还有其他选择:

1. 您可以研究使用内存映射文件.

1. You could look into using memory mapped files.

如果数组中包含很多零，则可以将其存储为稀疏矩阵表示形式(不要显式存储0).

If the array has a lot of zeros in it, you could store it as a sparse matrix representation (don't explicitly store 0s).

如果您不需要立即将所有内容都存储在内存中，则还可以将其存储在某种持久性存储中(文件，数据库等)，并且仅在任何给定时间访问所需的部分.

If you don't need the whole thing in memory at once, you could also store it in some sort of persistent storage (file, database, etc) and only access the parts you need at any given time.

这篇关于Java中的大数组导致的内存不足异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！