如何为 TensorFlow Estimators 充分利用 GPU? [英] How to make best use of GPU for TensorFlow Estimators?

问题描述

我的深度学习模型使用的是 Tensorflow(CPU 版本).具体使用 DNNRegressor Estimator 进行训练，使用给定的参数集(网络结构、隐藏层、alpha 等)虽然我能够减少损失，但模型需要非常长的学习时间(大约 3 天.)和时间每 100 步需要 9 秒.

I was using Tensorflow(CPU version) for my Deep Learning Model. Specifically using DNNRegressor Estimator for training, with given set of parameters (network structure, hidden layers, alpha etc.) Though I was able to reduce the loss, but model took very large time for learning (approx 3 days.) and time it was taking was 9 sec per 100th step.

我看到了这篇文章:- 使用 4(vCPU)、12(ECU) 和 61 (MiB).

I came accross this article :- https://medium.com/towards-data-science/how-to-traine-tensorflow-models-79426dabd304 and found that GPU's can be more faster to learn. So i took p2.xlarge gpu from AWS (single core GPU) with 4(vCPU), 12(ECU) and 61 (MiB).

但是学习率相同，每 100 步 9 秒.我在 CPU 上使用与 Estimator 相同的代码，因为我读到 Estimator 自己使用 GPU.这是我的nvidia-smi"命令输出.

But the learning rate is same at 9 sec per 100th step. I m using same code I used for Estimators on CPU, because I read that Estimators use GPU on their own. Here is my "nvidia-smi" command output.

• 显示正在使用 GPU 内存，但我的 Volatile GPU-Util 为 1%.无法弄清楚，我错过了什么.它的设计目的是相同的还是我遗漏了什么，因为对于 Estimator 的 CPU 和 GPU 实现，每秒全局步数是相同的.
• 我是否必须明确更改 DNNRegressor Estimator 代码中的某些内容?

推荐答案

听起来您可能正在从 csv 读取并转换为 pandas DataFrame，然后使用 tensorflow 的 pandas_input_fn.这是 pandas_input_fn 实现的一个已知问题.您可以在 https://github.com/tensorflow/tensorflow/issues/13530.

It sounds like you might be reading from csv and converting to a pandas DataFrame, and then using tensorflow's pandas_input_fn. This is a known issue with the implementation of pandas_input_fn. You can track the issue at https://github.com/tensorflow/tensorflow/issues/13530.

为了解决这个问题，您可以对 i/o 使用不同的方法(例如，从 TFRecords 读取).如果您想继续使用 Pandas 并增加您的步数/秒，您可以减少您的 batch_size，尽管这可能会对您的估算器的学习能力产生负面影响.

To combat this, you can use a different method for i/o (reading from TFRecords, for example). If you'd like to continue using pandas and increase your steps/second, you can reduce your batch_size, though this may have negative consequences on your estimator's ability to learn.

这篇关于如何为 TensorFlow Estimators 充分利用 GPU?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！