是否可以在 tensorflow 中跨多个 GPU 拆分网络? [英] Is it possible to split a network across multiple GPUs in tensorflow?
问题描述
我计划运行一个非常大的循环网络(例如 2048x5),是否可以在 tensorflow 中的一个 GPU 上定义一层?我应该如何实现模型以达到最佳效率.我知道 GPU 间或 GPU-CPU-GPU 通信存在开销.
I plan to run a very large recurrent network (e.g. 2048x5), is it possible to define one layer at one GPU in tensorflow? How should I implement the model to achieve the best efficiency. I understand there is overhead for inter-GPU or GPU-CPU-GPU communication.
推荐答案
在 TensorFlow 中跨多个 GPU 拆分大型模型当然是可能的,但以最佳方式实现它是一个困难的研究问题.通常,您需要执行以下操作:
Splitting a large model across multiple GPUs is certainly possible in TensorFlow, but doing it optimally is a hard research problem. In general, you will need to do the following:
用
包装你的代码的大面积连续区域使用 tf.device(...):
块,命名不同的 GPU:
Wrap large contiguous regions of your code in a
with tf.device(...):
block, naming the different GPUs:
with tf.device("/gpu:0"):
# Define first layer.
with tf.device("/gpu:1"):
# Define second layer.
# Define other layers, etc.
在构建优化器时,将可选参数 colocate_gradients_with_ops=True
传递给 optimizer.minimize()
方法:
When building your optimizer, pass the optional argument colocate_gradients_with_ops=True
to the optimizer.minimize()
method:
loss = ...
optimizer = tf.train.AdaGradOptimizer(0.01)
train_op = optimizer.minimize(loss, colocate_gradients_with_ops=True)
(可选.)在创建 tf.Session
,如果您模型中的任何操作无法在 GPU 上运行:
(Optionally.) You may need to enable "soft placement" in the tf.ConfigProto
when you create your tf.Session
, if any of the operations in your model cannot run on GPU:
config = tf.ConfigProto(allow_soft_placement=True)
sess = tf.Session(config=config)
这篇关于是否可以在 tensorflow 中跨多个 GPU 拆分网络?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!