Caffe:如果只有一小部分内存可以怎么办? [英] Caffe: What can I do if only a small batch fits into memory?

问题描述

我正在尝试训练一个非常大的模型.因此，我只能将非常小的批处理大小放入GPU内存中.使用小批量生产的结果非常嘈杂的梯度估计.
我该怎么做才能避免这个问题?

I am trying to train a very large model. Therefore, I can only fit a very small batch size into GPU memory. Working with small batch sizes results with very noisy gradient estimations.
What can I do to avoid this problem?

推荐答案

您可以在求解器参数中更改iter_size. Caffe在每个随机梯度下降步骤中的iter_size x batch_size实例上累积梯度. 因此，当由于内存有限而无法使用大的batch_size时，增加iter_size也会获得更稳定的渐变.

You can change the iter_size in the solver parameters. Caffe accumulates gradients over iter_size x batch_size instances in each stochastic gradient descent step. So increasing iter_size can also get more stable gradient when you cannot use large batch_size due to the limited memory.

这篇关于Caffe:如果只有一小部分内存可以怎么办?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！