当列表项失败或花费时间太长时，我如何使lapply超时? [英] How do I time out a lapply when a list item fails or takes too long?

问题描述

通过目前的工作，我正在通过一系列功能来运行具有大量参数组合的大型数据集.这些函数具有包装器(因此我可以mclapply)，以简化在群集上的操作.但是，我遇到了两个主要挑战.

For several efforts I'm involved in at the moment, I am running large datasets with numerous parameter combinations through a series of functions. The functions have a wrapper (so I can mclapply) for ease of operation on a cluster. However, I run into two major challenges.

a)我的参数组合很大(想想20k到100k).有时某些特定组合会会失败(例如，生存率太高而死亡率太低，因此该模型永远不会收敛为假设情况).对于我来说，很难提前准确地确定哪些组合会失败(如果我能够做到，生活会更容易).但是现在我有这种类型的设置:

a) My parameter combinations are large (think 20k to 100k). Sometimes particular combinations will fail (e.g. survival is too high and mortality is too low so the model never converges as a hypothetical scenario). It's difficult for me to suss out ahead of time exactly which combinations will fail (life would be easier if I could do that). But for now I have this type of setup:

failsafe <- failwith(NULL, my_wrapper_function)
# This is what I run
# Note that input_variables contains a list of variables in each list item
results <-  mclapply(input_variables, failsafe, mc.cores = 72)
# On my local dual core mac, I can't do this so the equivalent would be:
results <-  llply(input_variables, failsafe,  .progress = 'text')

包装函数的框架如下:

my_wrapper_function <- function(tlist) {
    run <- tryCatch(my_model(tlist$a, tlist$b, tlist$sA, tlist$Fec, m = NULL) , error=function(e) NULL)
...
return(run)
}

这是最有效的方法吗?如果由于某种原因，特定的变量组合使模型崩溃，我需要它返回NULL并继续使用其余的模型.但是，我仍然遇到这样的问题，即失败的程度不止于此.

Is this the most efficient approach? If for some reason a particular combination of variables crashes the model, I need it to return a NULL and carry on with the rest. However, I still have issues that this fails less than gracefully.

b)有时某些输入组合不会使模型崩溃，但是收敛所需的时间太长.我对群集上的计算时间设置了限制(例如6个小时)，因此我不会将资源浪费在被卡住的东西上.如何包含一个超时，以便如果一个函数调用在单个列表项上花费的时间超过x次，则该超时应该继续进行?计算花费的时间是微不足道的，但是不能中断模拟中的功能来检查时间，对吧?

b) Sometimes a certain combination of inputs does not crash the model but takes too long to converge. I set a limit on the computation time on my cluster (say 6 hours) so I don't waste my resources on something that is stuck. How can I include a timeout such that if a function call takes more than x time on a single list item, it should move on? Calculating the time spent is trivial but a function mid simulation can't be interrupted to check the time, right?

任何想法，解决方案或技巧都会受到赞赏！

Any ideas, solutions or tricks are appreciated!

推荐答案

您可能能够使用R.utils软件包中的tryCatch()和evalWithTimeout()的组合来管理优美退出时机. 另请参阅此帖子，它显示了类似的代码，并且对其进行了更详细的解压缩.

You may well be able to manage graceful-exits-upon-timout using a combination of tryCatch() and evalWithTimeout() from the R.utils package. See also this post, which presents similar code and unpacks it in a bit more detail.

require(R.utils)

myFun <- function(x) {Sys.sleep(x); x^2}

## evalWithTimeout() times out evaluation after 3.1 seconds, and then
## tryCatch() handles the resulting error (of class "TimeoutException") with 
## grace and aplomb.
myWrapperFunction <- function(i) {
    tryCatch(expr = evalWithTimeout(myFun(i), timeout = 3.1), 
             TimeoutException = function(ex) "TimedOut")
}

sapply(1:5, myWrapperFunction)
# [1] "1"        "4"        "9"        "TimedOut" "TimedOut"

这篇关于当列表项失败或花费时间太长时，我如何使lapply超时?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！