apply()很慢-如何使其更快，或者我有什么选择? [英] apply() is slow - how to make it faster or what are my alternatives?

问题描述

我有一个很大的数据框，大约有一千万行.它具有列x和y，我要计算的是

I have a quite large data frame, about 10 millions of rows. It has columns x and y, and what I want is to compute

hypot <- function(x) {sqrt(x[1]^2 + x[2]^2)}

每行

.使用apply会花费很多时间(大约5分钟，从较小的尺寸进行插值)和内存.

for each row. Using apply it would take a lot of time (about 5 minutes, interpolating from lower sizes) and memory.

但这对我来说似乎太多了，所以我尝试了不同的方法:

But it seems to be too much for me, so I've tried different things:

• 编译hypot函数可将时间减少约10％
• 使用plyr中的功能会大大增加运行时间.

• compiling the hypot function reduces the time by about 10%
• using functions from plyr greatly increases the running time.

做这件事的最快方法是什么?

What's the fastest way to do this thing?

推荐答案

with(my_data,sqrt(x^2+y^2))怎么样?

set.seed(101)
d <- data.frame(x=runif(1e5),y=runif(1e5))

library(rbenchmark)

每行有两种不同的功能，一种是利用矢量化的功能:

Two different per-line functions, one taking advantage of vectorization:

hypot <- function(x) sqrt(x[1]^2+x[2]^2)
hypot2 <- function(x) sqrt(sum(x^2))

也尝试编译它们:

library(compiler)
chypot <- cmpfun(hypot)
chypot2 <- cmpfun(hypot2)

benchmark(sqrt(d[,1]^2+d[,2]^2),
          with(d,sqrt(x^2+y^2)),
          apply(d,1,hypot),
          apply(d,1,hypot2),
          apply(d,1,chypot),
          apply(d,1,chypot2),
          replications=50)

结果:

                       test replications elapsed relative user.self sys.self
5       apply(d, 1, chypot)           50  61.147  244.588    60.480    0.172
6      apply(d, 1, chypot2)           50  33.971  135.884    33.658    0.172
3        apply(d, 1, hypot)           50  63.920  255.680    63.308    0.364
4       apply(d, 1, hypot2)           50  36.657  146.628    36.218    0.260
1 sqrt(d[, 1]^2 + d[, 2]^2)           50   0.265    1.060     0.124    0.144
2  with(d, sqrt(x^2 + y^2))           50   0.250    1.000     0.100    0.144

如预期的那样，with()解决方案和la Tyler Rinker色谱柱索引解决方案基本相同； hypot2的速度是原始hypot的两倍(但仍比矢量化解决方案慢约150倍).正如OP所指出的那样，编译并没有太大帮助.

As expected the with() solution and the column-indexing solution à la Tyler Rinker are essentially identical; hypot2 is twice as fast as the original hypot (but still about 150 times slower than the vectorized solutions). As already pointed out by the OP, compilation doesn't help very much.

这篇关于apply()很慢-如何使其更快，或者我有什么选择?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！