ReLU没有学习处理负输入Keras/Tensorflow [英] ReLU not learning to handle negative inputs Keras / Tensorflow

问题描述

我希望我的神经网络将负值转换为正值.从理论上讲，这可以使用ReLU函数和1个节点来完成，输入输入权重为-1(因此，负输入乘以-1 =正输入.

I want my neural network to convert a negative value into a positive value. Theoretically this can be done using a ReLU function and 1 node which learns the input weight to be -1 (so a negative input is multiplied by -1 = positive input.

它只会继续输出0.下面的代码.我使用-1作为输入值，以查看它是否至少可以在单个输入上学习.

It just keeps on outputting 0. Code below. I used -1 as input values to see if it could learn on at least a single input.

我尝试添加更多的图层，但无济于事请参见编辑，如果我添加更多的内容，则DIT帮助

I tried adding more layers but it doesn't help see edit, IT DID help if I add more

train_input = np.asarray([[-1]]*10000) # Input arr of -1s
train_output = np.asarray(map(lambda x: [abs(x[0])] , train_input))

# Define the model
model = Sequential()
model.add(Dense(1, input_dim=1, kernel_initializer='normal', activation='linear'))
model.add(LeakyReLU(alpha=.001))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

# Train and evaluate
model.fit(train_input, train_output, epochs=10, batch_size=10, verbose=0)
test_model_output = model.predict(test_input)
print str(test_input[0][0]) + " " + str(test_output[0][0]) + " " +  str(test_model_output[0][0])

我得到的输出低于(第一个输入值，第二个是预期输出，第三个是模型输出)

The output I get is below (1st value is input, 2nd is expected output, 3rd is model output )

-1 1 0.0

编辑 我尝试使用随机统一初始化程序，这样可以初始化负权重，并且可以正常工作.我明白为什么这应该使网络学习变得更轻松.但是我不明白为什么有必要.

EDIT I tried using the random uniform initialiser so it would initialise negative weights and it works. I get why this should make it easier for the network to learn. But I don't get why it's necessary.

from keras.initializers import RandomUniform
model.add(Dense(1, input_dim=1, kernel_initializer=RandomUniform(minval=-0.05, maxval=0.05, seed=None), activation='linear'))

编辑2 有人提到我没有足够的时间来训练数据.最初，我认为将数据增加10倍，将批次减小10倍(更多迭代)是可行的.没有 但是，如果我添加了10倍以上的时间(总共100个)，它确实起作用了.因此，将正的初始权重转换为负的时间很长

EDIT 2 Someone mentioned I didn't have enough time to train the data. At first I thought making it 10x more data and batches to be 10x smaller (more iterations) would work. It didn't BUT if I added 10x more epochs (100 total) it did work. So it just takes a long time to convert positive initialised weights to negative

推荐答案

问题是我没有给它足够的培训时间.虽然这是一个非常基本的功能，但初始化的权重必须从负数变为正数.

The problem was that I was not giving it enough time to train. While it's a very basic function the initialised weights have to go form negative to positive.

增加训练量(时间越长，批次越少，训练数据越多)，最终导致梯度从正值变为负值.

Increase the amount of training done (more epochs, smaller batches, more training data) eventually caused the gradient to flip from positive to negative.

这篇关于ReLU没有学习处理负输入Keras/Tensorflow的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！