使用以前的计算值(滚动)时最有效/向量化 [英] Most efficient/vectorization when using previous calculated value (rolling)
本文介绍了使用以前的计算值(滚动)时最有效/向量化的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
在这些会话后:
- Can I vectorize a calculation which depends on previous elements
- sapply? tapply? ddply? dataframe variable based on rolling index of previous values of another variable
我想测试一个更现实生活案例分析。
我最近不得不将SAS代码迁移到R,将kdb代码迁移到R代码。
I wanted to test a more "real-life" case-study. I recently had to migrate SAS code to R and kdb code to R code. I tried to compiled a simple enough yet more sophisticated example to optimize.
让我们建立训练集
buildDF <- function(N){
set.seed(123); dateTimes <- sort(as.POSIXct("2001-01-01 08:30:00") + floor(3600*runif(N)));
set.seed(124); f <- floor(1+3*runif(N));
set.seed(123); s <- floor(1+3*runif(N));
return(data.frame(dateTime=dateTimes, f=f, s=s));
}
这是需要实现的
f1 <- function(DF){
#init
N <- nrow(DF);
DF$num[1] = 1;
for(i in 2:N){
if(DF$f[i] == 2){
DF$num[i] <- ifelse(DF$s[i-1] == DF$s[i],DF$num[i-1],1+DF$num[i-1]);
}else{ #meaning f in {1,3}
if(DF$f[i-1] != 2){
DF$num[i] = DF$num[i-1];
}else{
DF$num[i] = ifelse((DF$dateTime[i]-DF$dateTime[i-1])==0,DF$num[i-1],1+DF$num[i-1]);
}
}
}
return(DF)
}
这是不可取的课程。让我们将它向量化一下:
This is off course hideous. Let's vectorize it a bit:
f2 <- function(DF){
N <- nrow(DF);
DF$add <- 1; DF$ds <- c(NA,diff(DF$s)); DF$lf <- c(NA,DF$f[1:(N-1)]);
DF$dt <- c(NA,diff(DF$dateTime));
DF$add[DF$f == 2 & DF$ds == 0] <- 0;
DF$add[DF$f == 2 & DF$ds != 0] <- 1;
DF$add[DF$f != 2 & DF$lf != 2] <- 0;
DF$add[DF$f != 2 & DF$lf == 2 & DF$dt==0] <- 0;
DF$num <- cumsum(DF$add);
return(DF);
}
并使用最有用的 data.table :
f3 <- function(DT){
N <- nrow(DT);
DT[,add:=1]; DT[,ds:=c(NA,diff(s))]; DT[,lf:=c(NA,f[1:(N-1)])];
DT[,dt:=c(NA,diff(dateTime))];
DT[f == 2 & ds == 0, add:=0];
DT[f == 2 & ds != 0, add:=1];
DT[f != 2 & lf != 2, add:=0];
DT[f != 2 & lf == 2 & dt == 0, add:=0];
DT[,num:=cumsum(add)];
return(DT);
}
在10K数据框上:
library(rbenchmark);
library(data.table);
N <- 1e4;
DF <- buildDF(N)
DT <- as.data.table(DF);#we can contruct the data.table as a data.frame so it's ok we don't count for this time.
#make sure everybody is equal
DF1 <- f1(DF) ; DF2 <- f2(DF); DT3 <- f3(DT);
identical(DF1$num,DF2$num,DT3$num)
[1] TRUE
#let's benchmark
benchmark(f1(DF),f2(DF),f3(DT),columns=c("test", "replications", "elapsed",
+ "relative", "user.self", "sys.self"), order="relative",replications=1);
test replications elapsed relative user.self sys.self
2 f2(DF) 1 0.010 1.0 0.012 0.000
3 f3(DT) 1 0.012 1.2 0.012 0.000
1 f1(DF) 1 9.085 908.5 8.980 0.072
好吧,现在在一个更体面的5M行data.frame
Ok, now on a more decent 5M rows data.frame
N <- 5e6;
DF <- buildDF(N)
DT <- as.data.table(DF);
benchmark(f2(DF),f3(DT),columns=c("test", "replications", "elapsed",
+ "relative", "user.self", "sys.self"), order="relative",replications=1);
test replications elapsed relative user.self sys.self
2 f3(DT) 1 2.843 1.000 2.092 0.624
1 f2(DF) 1 10.920 3.841 4.016 5.137
我们用data.table获得了5X。
We gain 5X with data.table.
Rcpp 或动物园::: rollapply可以在此获得更多。
我会满意任何建议
I wonder if Rcpp or zoo:::rollapply can gain much on this. I would be happy with any suggestion
推荐答案
简单内联Rcpp版本:
Simple inline Rcpp version:
library(Rcpp)
library(inline)
f4cxx <- cxxfunction(signature(input="data.frame"), plugin="Rcpp", body='
Rcpp::DataFrame df(input);
const int N = df.nrows();
Rcpp::NumericVector f = df["f"];
Rcpp::NumericVector s = df["s"];
Rcpp::NumericVector d = df["dateTime"]; // As far as we need only comparation
Rcpp::NumericVector num(N); // it is safe to convert Datetime to Numeric (faster)
num[0] = 1;
for(int i=1; i<N; i++){
bool cond1 = (f[i]==2) && (s[i]!=s[i-1]);
bool cond2 = (f[i]!=2) && (f[i-1]==2) && (d[i]!=d[i-1]);
num[i] = (cond1 || cond2)?1+num[i-1]:num[i-1];
}
df["num"] = num;
return df; // Returns list
//return (Rcpp::as<Rcpp::DataFrame>(df)); // Returns data.frame (slower)
')
签出:
N<-1e4; df<-buildDF(N)
identical(f1(df)$num, f4cxx(df)$num)
[1] TRUE
基准:
N<-1e5; df<-buildDF(N); dt<-as.data.table(df)
benchmark(f2(df), f2j(df), f3(dt), f4cxx(df),
columns=c("test", "replications", "elapsed", "relative", "user.self", "sys.self"),
order="relative", replications=1);
test replications elapsed relative user.self sys.self
4 f4cxx(df) 1 0.001 1 0.000 0
2 f2j(df) 1 0.037 37 0.040 0
3 f3(dt) 1 0.058 58 0.056 0
1 f2(df) 1 0.078 78 0.076 0
这篇关于使用以前的计算值(滚动)时最有效/向量化的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文