我是用plyr吗？我似乎使用太多的内存方式 [英] Am I using plyr right? I seem to be using way too much memory

问题描述

我有以下，有点大的数据集：

I have the following, somewhat large dataset:

 > dim(dset)
 [1] 422105     25
 > class(dset)
 [1] "data.frame"
 > 

不做任何事情，R进程似乎需要大约1GB的RAM。

Without doing anything, the R process seems to take about 1GB of RAM.

我试图运行以下代码：

  dset <- ddply(dset, .(tic), transform,
                date.min <- min(date),
                date.max <- max(date),
                daterange <- max(date) - min(date),
                .parallel = TRUE)

那个代码，RAM使用天空。它完全饱和了60GB的RAM，运行在32核机器上。我做错了什么？

Running that code, RAM usage skyrockets. It completely saturated 60GB's of RAM, running on a 32 core machine. What am I doing wrong?

推荐答案

如果性能是一个问题，最好切换到使用 data.table s从同名的包。他们是快速。你可以做一些大致相当于这样的东西：

If performance is an issue, it might be a good idea to switch to using data.tables from the package of the same name. They are fast. You'd do something roughly equivalent something like this:

library(data.table)
dat <- data.frame(x = runif(100),
                  dt = seq.Date(as.Date('2010-01-01'),as.Date('2011-01-01'),length.out = 100),
                  grp = rep(letters[1:4],each = 25))

dt <- as.data.table(dat)
key(dt) <- "grp"

dt[,mutate(.SD,date.min = min(dt),
               date.max = max(dt),
               daterange = max(dt) - min(dt)), by = grp]

这篇关于我是用plyr吗？我似乎使用太多的内存方式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！