在RNN& amp;中使用批处理规范化是否正常?LSTM? [英] Is it normal to use batch normalization in RNN & LSTM?

问题描述

我知道在常规的神经网络中，人们会在激活之前使用批处理规范，这将减少对良好的权重初始化的依赖.我想知道当我使用它时是否会对RNN/lstm RNN起到同样的作用.有人有经验吗?

I know in regular neural nets people use batch norm before activation and it will reduce the reliance on good weight initialization. I wonder if it would do the same to RNN/lstm RNN when i use it. Does anyone have any experience with it?

推荐答案

不，您不能在循环神经网络上使用批量归一化，因为统计数据是按批次计算的，因此不考虑网络的循环部分.权重在RNN中共享，每个循环"的激活响应可能具有完全不同的统计属性.

No, you cannot use Batch Normalization on a recurrent neural network, as the statistics are computed per batch, this does not consider the recurrent part of the network. Weights are shared in an RNN, and the activation response for each "recurrent loop" might have completely different statistical properties.

已经开发出了与批处理标准化类似的其他技术，其中考虑了这些限制，例如图层标准化.LSTM层也有重新设置参数，可以使用批处理规范化，例如，Coijmaans等人在经常批处理规范化中进行了描述al.2016.

Other techniques similar to Batch Normalization that take these limitations into account have been developed, for example Layer Normalization. There are also reparametrizations of the LSTM layer that allow Batch Normalization to be used, for example as described in Recurrent Batch Normalization by Coijmaans et al. 2016.

这篇关于在RNN& amp;中使用批处理规范化是否正常?LSTM?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！