与占用大量内存的列表对象进行交互工作 [英] Interactively work with list objects that take up massive memory

问题描述

我最近发现了bigmemory，ff和filehash软件包用于处理超大型矩阵的奇妙之处.

I have recently discovered the wonders of the packages bigmemory, ff and filehash to handle very large matrices.

如何处理很大的列表(300MB ++)?在我的工作中，我每天都每天使用这些列表.我可以使用save()&做创可贴解决方案load()到处都是骇客，但我更喜欢类似bigmemory的解决方案.像bigmemory bigmatrix之类的东西将是理想的，我在其中使用与matrix基本上相同的东西，只是它占用了RAM中的660字节之类的东西.

How can I handle very large (300MB++) lists? In my work I work with these lists all day every day. I can do band-aid solution with save() & load() hacks everywhere but I would prefer a bigmemory-like solution. Something like a bigmemory bigmatrix would be ideal, where I work with it basically identically to a matrix except it takes up somethign like 660 bytes in my RAM.

这些列表主要是lm()个对象(或类似回归对象)的>1000个长度列表.例如，

These lists are mostly >1000 length lists of lm() objects (or similar regression objects). For example,

Y <- rnorm(1000) ; X <- rnorm(1000)
A <- lapply(1:6000, function(i) lm(Y~X))
B <- lapply(1:6000, function(i) lm(Y~X))
C <- lapply(1:6000, function(i) lm(Y~X))
D <- lapply(1:6000, function(i) lm(Y~X))
E <- lapply(1:6000, function(i) lm(Y~X))
F <- lapply(1:6000, function(i) lm(Y~X))

在我的项目中，我将拥有A,B,C,D,E,F类型的列表(甚至更多)，这些列表必须进行交互处理.

In my project I will have A,B,C,D,E,F-type lists (and even more than this) that I have to work with interactively.

如果这些是巨大的矩阵，则有大量的支持.我想知道在大型list对象的任何软件包中是否有类似的支持.

If these were gigantic matrices there is a tonne of support. I was wondering if there was any similar support in any package for large list objects.

推荐答案

您可以使用 filehash 软件包.这应该可以工作(如果在我的机器上速度很慢...):

You can store and access lists on disk using the filehash package. This should work (if rather slowly on my machine...):

Y <- rnorm(1000) ; X <- rnorm(1000)

# set up disk object
library(filehash)
dbCreate("myTestDB")
db <- dbInit("myTestDB")

db$A <- lapply(1:6000, function(i) lm(Y~X))
db$B <- lapply(1:6000, function(i) lm(Y~X))
db$C <- lapply(1:6000, function(i) lm(Y~X))
db$D <- lapply(1:6000, function(i) lm(Y~X))
db$E <- lapply(1:6000, function(i) lm(Y~X))
db$F <- lapply(1:6000, function(i) lm(Y~X))

可以使用[功能访问列表项.有关更多详细信息，请参见此处: http://cran.r-project .org/web/packages/filehash/vignettes/filehash.pdf

List items can be accessed using the [ function. See here for more details: http://cran.r-project.org/web/packages/filehash/vignettes/filehash.pdf

这篇关于与占用大量内存的列表对象进行交互工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！