我可以使用`tf.nn.dropout`来实现DropConnect吗? [英] Can I use `tf.nn.dropout` to implement DropConnect?

问题描述

我(认为)我掌握了DropOut的基础知识以及实现TensorFlow API的用法..但是与tf.nn.dropout中的辍学概率相关的归一化似乎不是 DropConnect .那是对的吗?如果是这样，归一化是否有任何危害"，或者我可以简单地对权重应用tf.nn.dropout来实现DropConnect吗?

I (think) that I grasp the basics of DropOut and the use of the TensorFlow API in implementing it. But the normalization that's linked to the dropout probability in tf.nn.dropout seems not to be a part of DropConnect. Is that correct? If so, does normalizing do any "harm" or can I simply apply tf.nn.dropout to my weights to implement DropConnect?

推荐答案

答案

是的，您可以使用 tf.nn.dropout 进行 DropConnect ，只需使用 tf.nn.dropout 包装体重矩阵而不是矩阵矩阵乘法.然后您可以像这样乘以辍学来撤消体重变化

Answer

Yes, you can use tf.nn.dropout to do DropConnect, just use tf.nn.dropout to wrap your weight matrix instead of your post matrix multiplication. You can then undo the weight change by multiplying by the dropout like so

dropConnect = tf.nn.dropout( m1, keep_prob ) * keep_prob

代码示例

这里是一个代码示例，该示例使用拖放连接计算 XOR 函数.我还注释掉了可以删除的代码，您可以将其包含在其中并比较输出.

Code Example

Here is a code example that calculates the XOR function using drop connect. I've also commented out the code that does dropout that you can sub in and compare the output.

### imports
import tensorflow as tf

### constant data
x  = [[0.,0.],[1.,1.],[1.,0.],[0.,1.]]
y_ = [[1.,0.],[1.,0.],[0.,1.],[0.,1.]]

### induction

# Layer 0 = the x2 inputs
x0 = tf.constant( x  , dtype=tf.float32 )
y0 = tf.constant( y_ , dtype=tf.float32 )

keep_prob = tf.placeholder( dtype=tf.float32 )

# Layer 1 = the 2x12 hidden sigmoid
m1 = tf.Variable( tf.random_uniform( [2,12] , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
b1 = tf.Variable( tf.random_uniform( [12]   , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))


########## DROP CONNECT
# - use this to preform "DropConnect" flavor of dropout
dropConnect = tf.nn.dropout( m1, keep_prob ) * keep_prob
h1 = tf.sigmoid( tf.matmul( x0, dropConnect ) + b1 ) 

########## DROP OUT
# - uncomment this to use "regular" dropout
#h1 = tf.nn.dropout( tf.sigmoid( tf.matmul( x0,m1 ) + b1 ) , keep_prob )


# Layer 2 = the 12x2 softmax output
m2 = tf.Variable( tf.random_uniform( [12,2] , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
b2 = tf.Variable( tf.random_uniform( [2]   , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
y_out = tf.nn.softmax( tf.matmul( h1,m2 ) + b2 )


# loss : sum of the squares of y0 - y_out
loss = tf.reduce_sum( tf.square( y0 - y_out ) )

# training step : discovered learning rate of 1e-2 through experimentation
train = tf.train.AdamOptimizer(1e-2).minimize(loss)

### training
# run 5000 times using all the X and Y
# print out the loss and any other interesting info
with tf.Session() as sess:
  sess.run( tf.initialize_all_variables() )
  print "\nloss"
  for step in range(5000) :
    sess.run(train,feed_dict={keep_prob:0.5})
    if (step + 1) % 100 == 0 :
      print sess.run(loss,feed_dict={keep_prob:1.})


  results = sess.run([m1,b1,m2,b2,y_out,loss],feed_dict={keep_prob:1.})
  labels  = "m1,b1,m2,b2,y_out,loss".split(",")
  for label,result in zip(*(labels,results)) :
    print ""
    print label
    print result

print ""

输出

两种口味都能正确地将输入分离为正确的输出

Output

Both flavors are able to correctly separate the input into the correct output

y_out
[[  7.05891490e-01   2.94108540e-01]
 [  9.99605477e-01   3.94574134e-04]
 [  4.99370173e-02   9.50062990e-01]
 [  4.39682379e-02   9.56031740e-01]]

在这里您可以看到dropConnect的输出能够正确地将 Y 分类为true，true，false，false.

Here you can see the output from dropConnect was able to correctly classify Y as true,true,false,false.

这篇关于我可以使用`tf.nn.dropout`来实现DropConnect吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！