用神经网络和ReLU逼近正弦函数 [英] Approximating sine function with Neural Network and ReLU

问题描述

我正在尝试使用神经网络(Keras)近似正弦函数.

是的，我阅读了相关文章:)

• 为什么结果采用ReLU的形状?

  这与输出归一化有关吗?

  解决方案

  这里有两件事:

  1. 您的网络确实很浅且很小.只有四个带有 relu 的神经元使得有可能使其中的两个神经元完全完全饱和.这可能就是为什么您的网络结果看起来像这样的原因.尝试使用 he_normal 或 he_uniform 作为初始化程序来解决此问题.
  2. 我认为您的网络太小，无法完成此任务.我肯定会通过增加网络中更多的神经元和层来增加网络的深度和宽度.如果 sigmoid 具有与 sin 函数相似的形状，则可能会正常工作-但在 relu 的情况下，您确实需要更大的网络.

  I am trying to approximate a sine function with a neural network (Keras).

  Yes, I read the related posts :)

  • Link 1

  • Link 2

  • Link 3

  Using four hidden neurons with sigmoid and an output layer with linear activation works fine.

  But there are also settings that provide results that seem strange to me.

  Since I am just started to work with I am interested in what and why things happen, but I could not figure that out so far.

  # -*- coding: utf-8 -*-
  
  import numpy as np
  np.random.seed(7)
  
  from keras.models import Sequential
  from keras.layers import Dense
  import pylab as pl
  from sklearn.preprocessing import MinMaxScaler
  
  X = np.linspace(0.0 , 2.0 * np.pi, 10000).reshape(-1, 1)
  Y = np.sin(X)
  
  x_scaler = MinMaxScaler()
  #y_scaler = MinMaxScaler(feature_range=(-1.0, 1.0))
  y_scaler = MinMaxScaler()
  
  X = x_scaler.fit_transform(X)
  Y = y_scaler.fit_transform(Y)
  
  model = Sequential()
  model.add(Dense(4, input_dim=X.shape[1], kernel_initializer='uniform', activation='relu'))
  # model.add(Dense(4, input_dim=X.shape[1], kernel_initializer='uniform', activation='sigmoid'))
  # model.add(Dense(4, input_dim=X.shape[1], kernel_initializer='uniform', activation='tanh'))
  model.add(Dense(1, kernel_initializer='uniform', activation='linear'))
  
  model.compile(loss='mse', optimizer='adam', metrics=['mae'])
  
  model.fit(X, Y, epochs=500, batch_size=32, verbose=2)
  
  res = model.predict(X, batch_size=32)
  
  res_rscl = y_scaler.inverse_transform(res)
  
  Y_rscl = y_scaler.inverse_transform(Y)
  
  pl.subplot(211)
  pl.plot(res_rscl, label='ann')
  pl.plot(Y_rscl, label='train')
  pl.xlabel('#')
  pl.ylabel('value [arb.]')
  pl.legend()
  pl.subplot(212)
  pl.plot(Y_rscl - res_rscl, label='diff')
  pl.legend()
  pl.show()
  

  This is the result for four hidden neurons (ReLU) and linear output activation.

  Why does the result take the shape of the ReLU?

  Does this have something to do with the output normalization?

  解决方案

  Two things here:

  1. Your network is really shallow and small. Having only 4 neurons with relu makes a case when a couple of this neurons are completely saturated highly possible. This is probably why your network result looks like that. Try he_normal or he_uniform as initializer to overcome that.
  2. In my opinion your network is too small for this task. I would definitely increase both depth and width of your network by intdoucing more neurons and layers to your network. In case of sigmoid which has a similiar shape to a sin function this might work fine - but in case of relu you really need a bigger network.

  这篇关于用神经网络和ReLU逼近正弦函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！