如何将 Tensorflow 2.0 SavedModel 转换为 TensorRT? [英] How to convert Tensorflow 2.0 SavedModel to TensorRT?

问题描述

我已经在 Tensorflow 2.0 中训练了一个模型，并且正在尝试改进在转向生产时的预测时间(在具有 GPU 支持的服务器上).在 Tensorflow 1.x 中，我能够通过使用冻结图来获得预测加速，但这在 Tensorflow 2 中已被弃用.从阅读 Nvidia 对 TensorRT 的描述来看，他们建议使用 TensorRT 与单独使用 Tensorflow 相比，可以将推理速度提高 7 倍.来源:

I've trained a model in Tensorflow 2.0 and am trying to improve predict time when moving to production (on a server with GPU support). In Tensorflow 1.x I was able to get a predict speedup by using freeze graph, but this has been deprecated as of Tensorflow 2. From reading Nvidia's description of TensorRT, they suggest that using TensorRT can speedup inference by 7x compared to Tensorflow alone. Source:

TensorFlow 2.0 with TighterTensorRT 集成现已推出

我已经训练了我的模型并使用 Tensorflow 的 SavedModel 格式将其保存到一个 .h5 文件中.现在我按照 nvidia 的文档来优化模型以使用 tensorrt 进行推理:TF-TRT 2.0 工作流与 SavedModel.

I have trained my model and saved it to a .h5 file using Tensorflow's SavedModel format. Now I follow nvidia's documentation to optimize the model for inference with tensorrt: TF-TRT 2.0 Workflow With A SavedModel.

当我跑步时:

import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt

我收到错误:ModuleNotFoundError: No module named 'tensorflow.python.compiler.tensorrt'

他们在这里给出了 Tensorflow 2.0 的另一个例子:示例.但是，他们尝试导入与上述相同的模块，但我得到了同样的错误.

They give another example with Tensorflow 2.0 here: Examples. However, they try to import the same module as above and I get the same error.

谁能建议如何使用 TensorRT 优化我的模型?

Can anyone suggest how to optimize my model with TensorRT?

推荐答案

我已经解决了这个问题.问题是我在本地 Windows 机器上测试代码，而不是在支持 GPU 的 AWS EC2 实例上测试.

I've solved this issue. The problem is that I was testing the code on my local Windows machine, rather than on my AWS EC2 Instance with gpu support.

tensorflow.python.compiler.tensorrt 似乎包含在 tensorflow-gpu 中，但没有包含在标准 tensorflow 中.为了使用 TensorRT 转换 SavedModel 实例，您需要使用带有 tensorflow-gpu 的机器.(我知道这是运行模型所必需的，但没有意识到它是转换模型所必需的.)

It seems that tensorflow.python.compiler.tensorrt is included in tensorflow-gpu, but not in standard tensorflow. In order to convert the SavedModel instance with TensorRT, you need to use a machine with tensorflow-gpu. (I knew that this would be required to run the model, but hadn't realized it was needed to convert the model.)

这篇关于如何将 Tensorflow 2.0 SavedModel 转换为 TensorRT?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！