使用在 Mali GPU 上完全委托 BERT 模型 [英] Fully delegate BERT models on Mali GPU using

问题描述

我正在尝试使用 TensorflowLite 在 Mali GPU 上部署 BERT 和 Transformer 模型.但问题是TensorflowLite不支持这些模型中的一些操作，包括{CAST, GATHER, MUL, RESHAPE, UNPACK}.有谁知道我如何在 GPU 上委派这些操作?是否有任何其他 TensorflowLite 库可以支持嵌入式 GPU，特别是 Mali GPU?我只想测量它们在 GPU 上的延迟.

I am trying to deploy BERT and Transformer models on Mali GPU, using TensorflowLite. But the problem is that TensorflowLite does not support some operations in these models, including {CAST, GATHER, MUL, RESHAPE, UNPACK}. Does anyone have any idea how I can delegate those operations on GPU? Are there any other TensorflowLite libraries that could support embedded GPU and specifically Mali GPU? I just want to measure their latency on GPU.

STARTING!
Log parameter values verbosely: [0]
Min num runs: [1]
**Graph**: [mobilebert_float_384_gpu.tflite]
**Use gpu**: [1]
Loaded model mobilebert_float_384_gpu.tflite
**INFO**: Created TensorFlow Lite delegate for GPU.
**ERROR**: Following operations are not supported by GPU delegate:
CAST: Not supported cast case
GATHER: Operation is not supported.
MUL: MUL requires one tensor that not less than second in all dimensions.
RESHAPE: OP is supported, but tensor type isn't matched!
UNPACK: Operation is not supported.
**2661 operations will run on the GPU, and the remaining 81 operations will run on the CPU.**
INFO: Initialized OpenCL-based API.
INFO: Created 1 GPU delegate kernels.
Explicitly applied GPU delegate, and the model graph will be partially executed by the delegate w/ 1 delegate kernels.
The input model file size (MB): 100.239
Initialized session in 2491.9ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=2 first=434022 curr=247839 min=247839 max=434022 avg=340930 std=93091

推荐答案

GPU 委托不支持所有操作，仅支持 此列表.

The GPU delegate doesn't support all operations, just this list.

如果您可以使用 NNAPI 委托，您可能希望尝试一下.它可以将最佳"委托给不同的处理器，而不仅仅是 GPU.支持的操作是这里.

You may wish to try the NNAPI delegate if it's available to you. It can delegate to different processors as "best", not just the GPU. The supported operations are here.

最坏的情况，如果这些都不适合您，要么回退到 CPU，要么编写您自己的委托 - 但这是一大堆工作.

Worst case, if neither of those work for you is to either fallback to the CPU, or write your own delegate - but that's a chunk of work.

这篇关于使用在 Mali GPU 上完全委托 BERT 模型的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！