是否在R，整流线性单位(ReLU)激活功能中封装了“神经网络"? [英] Package ‘neuralnet’ in R, rectified linear unit (ReLU) activation function?

问题描述

我试图使用激活功能，而不是R包Neuronet中预先实现的逻辑"和"tanh".具体来说，我想使用整流线性单位(ReLU)f(x)= max {x，0}.请在下面查看我的代码.

I am trying to use activation functions other than the pre-implemented "logistic" and "tanh" in the R package neuralnet. Specifically, I would like to use rectified linear units (ReLU) f(x) = max{x,0}. Please see my code below.

我相信我可以使用自定义函数(例如，如果定义)

I believe I can use custom functions if defined by (for example)

custom <- function(a) {x*2}

但是如果我设置max(x，0)而不是x * 2，那么R会告诉我'max不在导数表中'，对于'>'运算符也是如此.因此，我正在寻找一个明智的解决方法，因为我认为在这种情况下，max的数值积分将不是问题.

but if I set max(x,0) instead of x*2 then R tells me that 'max is not in the derivatives table', and same for '>' operator. So I am looking for a sensible workaround as I am thinking numerical integration of max in this case wouldn't be an issue.

nn <- neuralnet(
  as.formula(paste("X",paste(names(Z[,2:10]), collapse="+"),sep="~")),
  data=Z[,1:10], hidden=5, err.fct="sse",
  act.fct="logistic", rep=1,
  linear.output=TRUE)

有什么想法吗?我有些困惑，因为我不认为neuralnet软件包不能进行分析差异.

Any ideas? I am a bit confused as I didn't think the neuralnet package would do analytical differentiation.

推荐答案

neuralnet包的内部将尝试区分提供给act.fct的任何功能.您可以在此处看到源代码.

The internals of the neuralnet package will try to differentiate any function provided to act.fct. You can see the source code here.

在211行，您将找到以下代码块:

At line 211 you will find the following code block:

if (is.function(act.fct)) {
    act.deriv.fct <- differentiate(act.fct)
    attr(act.fct, "type") <- "function"
}

differentiate函数是deriv函数的更复杂用法，您也可以在上面的源代码中看到.因此，当前无法将max(0,x)提供给act.fct.它将需要在代码中放置一个异常，以识别ReLU和知道派生类.获取源代码，将其添加进来并提交给维护人员以进行扩展(但这可能会很多)，将是一个很棒的练习.

The differentiate function is a more complex use of the deriv function which you can also see in the source code above. Therefore, it is currently not possible to provide max(0,x) to the act.fct. It would require an exception placed in the code to recognize the ReLU and know the derivative. It would be a great exercise to get the source code, add this in and submit to the maintainers to expand (but that may be a bit much).

但是，关于明智的解决方法，您可以使用 softplus函数是ReLU的平滑近似.您的自定义函数如下所示:

However, regarding a sensible workaround, you could use softplus function which is a smooth approximation of the ReLU. Your custom function would look like this:

custom <- function(x) {log(1+exp(x))}

您也可以在R中查看此近似值:

You can view this approximation in R as well:

softplus <- function(x) log(1+exp(x))
relu <- function(x) sapply(x, function(z) max(0,z))

x <- seq(from=-5, to=5, by=0.1)
library(ggplot2)
library(reshape2)

fits <- data.frame(x=x, softplus = softplus(x), relu = relu(x))
long <- melt(fits, id.vars="x")
ggplot(data=long, aes(x=x, y=value, group=variable, colour=variable))+
  geom_line(size=1) +
  ggtitle("ReLU & Softplus") +
  theme(plot.title = element_text(size = 26)) +
  theme(legend.title = element_blank()) +
  theme(legend.text = element_text(size = 18))

这篇关于是否在R，整流线性单位(ReLU)激活功能中封装了“神经网络"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！