Keras 中的 GaussianDropout 与 Dropout 与 GaussianNoise [英] GaussianDropout vs. Dropout vs. GaussianNoise in Keras

问题描述

谁能解释不同辍学风格之间的区别?从 文档 中，我假设不是删除一些单位为零(dropout)，GaussianDropout 将这些单位乘以某种分布.然而，在实际测试时，所有单元都被触及.结果看起来更像经典的 GaussianNoise.

Can anyone explain the difference between the different dropout styles? From the documentation, I assumed that instead of dropping some units to zero (dropout), GaussianDropout multiplies those units by some distribution. However, when testing in practice, all units are touched. The result looks more like the classic GaussianNoise.

tf.random.set_seed(0)
layer = tf.keras.layers.GaussianDropout(.05, input_shape=(2,))
data = np.arange(10).reshape(5, 2).astype(np.float32)
print(data)

outputs = layer(data, training=True)
print(outputs)

结果:

[[0. 1.]
 [2. 3.]
 [4. 5.]
 [6. 7.]
 [8. 9.]]
tf.Tensor(
[[0.    1.399]
 [1.771 2.533]
 [4.759 3.973]
 [5.562 5.94 ]
 [8.882 9.891]], shape=(5, 2), dtype=float32)

显然，这就是我一直想要的:

Apparently, this is what I wanted all along:

def RealGaussianDropout(x, rate, stddev):

    keep_prob = 1 - rate
    random_tensor = tf.random.uniform(tf.shape(x))
    keep_mask = tf.cast(random_tensor >= rate, tf.float32)   
    noised = x + K.random_normal(tf.shape(x), mean=.0, stddev=stddev)   
    ret = tf.multiply(x, keep_mask) + tf.multiply(noised, (1-keep_mask))

    return ret


outputs = RealGaussianDropout(data,0.2,0.1)
print(outputs)

推荐答案

你说得对... GaussianDropout 和 GaussianNoise 非常相似.您可以通过自己复制来测试所有相似之处

you are right... GaussianDropout and GaussianNoise are very similar. you can test all the similarities by reproducing them on your own

def dropout(x, rate):

    keep_prob = 1 - rate
    scale = 1 / keep_prob
    ret = tf.multiply(x, scale)
    random_tensor = tf.random.uniform(tf.shape(x))
    keep_mask = random_tensor >= rate
    ret = tf.multiply(ret, tf.cast(keep_mask, tf.float32))
    
    return ret

def gaussian_dropout(x, rate):
    
    stddev = np.sqrt(rate / (1.0 - rate))
    ret = x * K.random_normal(tf.shape(x), mean=1.0, stddev=stddev)
    
    return ret

def gaussian_noise(x, stddev):
    
    ret = x + K.random_normal(tf.shape(x), mean=.0, stddev=stddev)
    
    return ret

高斯噪声只是简单地将随机正态值与 0 均值相加，而高斯 dropout 只是将随机正态值与 1 均值相乘.这些操作涉及输入的所有元素.经典的 dropout 将一些输入元素变为 0 对其他元素进行缩放

Gaussian noise simply adds random normal values with 0 mean while gaussian dropout simply multiplies random normal values with 1 mean. These operations involve all the elements of the input. The classic dropout turn to 0 some input elements operating a scaling on the others

退出

data = np.arange(10).reshape(5, 2).astype(np.float32)

set_seed(0)
layer = tf.keras.layers.Dropout(.4)
out1 = layer(data, training=True)
set_seed(0)
out2 = dropout(data, .4)
print(tf.reduce_all(out1 == out2).numpy()) # TRUE

高斯下降

data = np.arange(10).reshape(5, 2).astype(np.float32)

set_seed(0)
layer = tf.keras.layers.GaussianDropout(.05)
out1 = layer(data, training=True)
set_seed(0)
out2 = gaussian_dropout(data, .05)
print(tf.reduce_all(out1 == out2).numpy()) # TRUE

GAUSSIANNOISE

data = np.arange(10).reshape(5, 2).astype(np.float32)

set_seed(0)
layer = tf.keras.layers.GaussianNoise(.3)
out1 = layer(data, training=True)
set_seed(0)
out2 = gaussian_noise(data, .3)
print(tf.reduce_all(out1 == out2).numpy()) # TRUE

为了保证我们使用的可重复性 (TF2):

to grant reproducibility we used (TF2):

def set_seed(seed):
    
    tf.random.set_seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    random.seed(seed)

这篇关于Keras 中的 GaussianDropout 与 Dropout 与 GaussianNoise的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！