tensorflow-大批量运行优化器op [英] tensorflow - run optimizer op on a large batch

问题描述

通常，我们以优化程序操作作为输入来调用运行命令，以更新某些模型的可训练参数:

Normally, we call the run command with the optimizer operation as input to update the trainable parameters of some model:

session.run(model.optimizer_op, feed_dict={model.X: X_batch, model.y: y_batch})

但是，当批处理量很大时，X_batch无法容纳在GPU内存中. 如何拆分此任务以处理大批量?

But when the batch size is large, X_batch can't fit in GPU memory. How can I split this task to handle a large batch size ?

推荐答案

这主要取决于您的GPU内存大小.但是，很难将整个数据集与模型及其所需的操作(即预测概率)一起拟合.因此，您将需要以不同的角度考虑批处理.我假设您的代码遵循以下原则:

This depends mainly on your GPU memory size. However, it is hard to fit your entire dataset along with the model and its required operations (i.e. predicting probabilities). Thus, you would need to think of batching in a different perspective. I assume your code goes along these lines:

# Model Definition    
X = tf.placeholder(tf.float32, shape=[None, DIM,DIM,3], name='X')
y = tf.placeholder(tf.float32, shape=[None, N_CLASSES], name='y')

...

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

...

# Training your model
sess.run([train_step], feed_dict={X: X_batch, y: y_batch})

而不是将X和y用作train_step的feed_dict.您可以为所有批次(即整个数据集)累积cross_entropy.然后，您可以运行一次train_step.例如:

Instead of using X and y as your feed_dict to the train_step. You could just accumulate the cross_entropy for all batches (i.e. for the entire dataset). Then, you can run the train_step once. For example:

cross_entropy_all = []
for X_batch, y_batch in batches_generator():
    cross_entropy_all += sess.run([cross_entropy], feed_dict={X: X_batch, y: y_batch})

# Numpy or Tensorflow equivalent for `vstack`
cross_entropy_all = np.vstack(cross_entropy_all)

# Run the optimizer on the entire dataset (not just on a specific batch)
sess.run([train_step], feed_dict={cross_entropy: cross_entropy_all})

这应该可以实现您的目标，而不会导致GPU内存不足.建议的方法针对所有交叉熵运行优化步骤.因此，您不需要输入X和y(用于生成cross_entropy的/必需的，因为它已经被输入到优化步骤中了.)

This should achieve your goal without running your GPU out of memory. The suggested approach runs the optimization step against all cross entropies. Thus, you don't need to feed X and y (that are used/needed to produce the cross_entropy because it is already fed to the optimization step).

这篇关于tensorflow-大批量运行优化器op的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！