R内存管理/无法分配大小为n Mb的向量 [英] R memory management / cannot allocate vector of size n Mb

问题描述

我在尝试在R中使用大对象时遇到了问题.例如:

I am running into issues trying to use large objects in R. For example:

> memory.limit(4000)
> a = matrix(NA, 1500000, 60)
> a = matrix(NA, 2500000, 60)
> a = matrix(NA, 3500000, 60)
Error: cannot allocate vector of size 801.1 Mb
> a = matrix(NA, 2500000, 60)
Error: cannot allocate vector of size 572.2 Mb # Can't go smaller anymore
> rm(list=ls(all=TRUE))
> a = matrix(NA, 3500000, 60) # Now it works
> b = matrix(NA, 3500000, 60)
Error: cannot allocate vector of size 801.1 Mb # But that is all there is room for

我了解这与获取连续的内存块有关(从

I understand that this is related to the difficulty of obtaining contiguous blocks of memory (from here):

错误消息开头不能 分配大小的向量表示一个 无法获取内存，或者 因为尺寸超出了 进程的地址空间限制，或者 更有可能是因为系统是 无法提供内存.笔记 在32位版本上可能 有足够的可用内存，但是 没有足够大的连续块 将其映射到的地址空间.

Error messages beginning cannot allocate vector of size indicate a failure to obtain memory, either because the size exceeded the address-space limit for a process or, more likely, because the system was unable to provide the memory. Note that on a 32-bit build there may well be enough free memory available, but not a large enough contiguous block of address space into which to map it.

我该如何解决?我的主要困难是我在脚本中到达了某个特定点，R无法为一个对象分配200-300 Mb ...我不能真正地预分配该块，因为我需要用于其他处理的内存.即使我认真删除不需要的对象，也会发生这种情况.

How can I get around this? My main difficulty is that I get to a certain point in my script and R can't allocate 200-300 Mb for an object... I can't really pre-allocate the block because I need the memory for other processing. This happens even when I dilligently remove unneeded objects.

是，对不起:Windows XP SP3、4Gb RAM，R 2.12.0:

Yes, sorry: Windows XP SP3, 4Gb RAM, R 2.12.0:

> sessionInfo()
R version 2.12.0 (2010-10-15)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_Caribbean.1252  LC_CTYPE=English_Caribbean.1252   
[3] LC_MONETARY=English_Caribbean.1252 LC_NUMERIC=C                      
[5] LC_TIME=English_Caribbean.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

推荐答案

考虑您是否确实确实需要所有这些数据，还是可以稀疏矩阵? R中的稀疏矩阵有很好的支持(例如，参见Matrix包).

Consider whether you really need all this data explicitly, or can the matrix be sparse? There is good support in R (see Matrix package for e.g.) for sparse matrices.

当您需要制作此大小的对象时，将R中的所有其他进程和对象保持在最低限度.使用gc()清除现在未使用的内存，或者更好的是仅在一个会话中创建所需的对象.

Keep all other processes and objects in R to a minimum when you need to make objects of this size. Use gc() to clear now unused memory, or, better only create the object you need in one session.

如果上述方法无济于事，请获取一台具有尽可能多RAM的64位计算机，然后安装64位R.

If the above cannot help, get a 64-bit machine with as much RAM as you can afford, and install 64-bit R.

如果您不能执行此操作，则可以使用许多在线服务进行远程计算.

If you cannot do that there are many online services for remote computing.

如果无法做到这一点，则内存映射工具(如软件包ff(或Sascha提到的bigmemory))将帮助您构建新的解决方案.以我有限的经验，ff是更高级的软件包，但是您应该阅读CRAN Task Views上的High Performance Computing主题.

If you cannot do that the memory-mapping tools like package ff (or bigmemory as Sascha mentions) will help you build a new solution. In my limited experience ff is the more advanced package, but you should read the High Performance Computing topic on CRAN Task Views.

这篇关于R内存管理/无法分配大小为n Mb的向量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！