apply() 很慢 - 如何使它更快或我的替代方案是什么? [英] apply() is slow - how to make it faster or what are my alternatives?

问题描述

我有一个相当大的数据框，大约有 1000 万行.它有 x 和 y 列，我想要的是计算

I have a quite large data frame, about 10 millions of rows. It has columns x and y, and what I want is to compute

hypot <- function(x) {sqrt(x[1]^2 + x[2]^2)}

对于每一行.使用 apply 会花费很多时间(大约 5 分钟，从较小的尺寸插入)和内存.

for each row. Using apply it would take a lot of time (about 5 minutes, interpolating from lower sizes) and memory.

但这对我来说似乎太多了，所以我尝试了不同的方法:

But it seems to be too much for me, so I've tried different things:

• 编译hypot函数可以减少大约10%的时间
• 使用 plyr 中的函数大大增加了运行时间.

• compiling the hypot function reduces the time by about 10%
• using functions from plyr greatly increases the running time.

做这件事最快的方法是什么?

What's the fastest way to do this thing?

推荐答案

with(my_data,sqrt(x^2+y^2)) 怎么样?

set.seed(101)
d <- data.frame(x=runif(1e5),y=runif(1e5))

library(rbenchmark)

两种不同的每行函数，一种利用矢量化:

Two different per-line functions, one taking advantage of vectorization:

hypot <- function(x) sqrt(x[1]^2+x[2]^2)
hypot2 <- function(x) sqrt(sum(x^2))

也尝试编译这些:

library(compiler)
chypot <- cmpfun(hypot)
chypot2 <- cmpfun(hypot2)

benchmark(sqrt(d[,1]^2+d[,2]^2),
          with(d,sqrt(x^2+y^2)),
          apply(d,1,hypot),
          apply(d,1,hypot2),
          apply(d,1,chypot),
          apply(d,1,chypot2),
          replications=50)

结果:

                       test replications elapsed relative user.self sys.self
5       apply(d, 1, chypot)           50  61.147  244.588    60.480    0.172
6      apply(d, 1, chypot2)           50  33.971  135.884    33.658    0.172
3        apply(d, 1, hypot)           50  63.920  255.680    63.308    0.364
4       apply(d, 1, hypot2)           50  36.657  146.628    36.218    0.260
1 sqrt(d[, 1]^2 + d[, 2]^2)           50   0.265    1.060     0.124    0.144
2  with(d, sqrt(x^2 + y^2))           50   0.250    1.000     0.100    0.144

正如预期的那样，with() 解决方案和 Tyler Rinker 的列索引解决方案本质上是相同的；hypot2 的速度是原始 hypot 的两倍(但仍然比矢量化解决方案慢 150 倍).正如 OP 已经指出的那样，编译并没有太大帮助.

As expected the with() solution and the column-indexing solution à la Tyler Rinker are essentially identical; hypot2 is twice as fast as the original hypot (but still about 150 times slower than the vectorized solutions). As already pointed out by the OP, compilation doesn't help very much.

这篇关于apply() 很慢 - 如何使它更快或我的替代方案是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！