Caffe:如果只有一小批适合内存，我该怎么办? [英] Caffe: What can I do if only a small batch fits into memory?

问题描述

我正在尝试训练一个非常大的模型.因此，我只能将非常小的批量装入 GPU 内存.使用小批量会导致非常嘈杂的梯度估计.
我该怎么做才能避免这个问题?

I am trying to train a very large model. Therefore, I can only fit a very small batch size into GPU memory. Working with small batch sizes results with very noisy gradient estimations.
What can I do to avoid this problem?

推荐答案

您可以更改求解器参数中的 iter_size.Caffe 在每个随机梯度下降步骤中累积 iter_size x batch_size 个实例的梯度.因此，当由于内存有限而无法使用大batch_size时，增加iter_size也可以获得更稳定的梯度.

You can change the iter_size in the solver parameters. Caffe accumulates gradients over iter_size x batch_size instances in each stochastic gradient descent step. So increasing iter_size can also get more stable gradient when you cannot use large batch_size due to the limited memory.

这篇关于Caffe:如果只有一小批适合内存，我该怎么办?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！