在 Tensorflow-lite Android 中将 Bitmap 转换为 ByteBuffer (float) [英] Converting Bitmap to ByteBuffer (float) in Tensorflow-lite Android

问题描述

在用于图像分类的 tensorflow-lite android 演示代码中，图像首先被转换为 ByteBuffer 格式以获得更好的性能.这种从位图到浮点格式的转换以及随后到字节缓冲区的转换似乎是一个昂贵的操作(循环，按位运算符，float mem-copy 等).我们试图用 opencv 实现相同的逻辑以获得一些速度优势.以下代码可以正常工作；但是由于这个转换中的一些逻辑错误，模型的输出(这个数据被馈送到)似乎是不正确的.模型的输入应该是数据类型为 float[1,197,197,3] 的 RGB.

我们如何使用 opencv(或任何其他方式)加速位图到字节缓冲区的转换过程?

标准位图到字节缓冲区的转换:-

/** 将图像数据写入 {@code ByteBuffer}.*/私有无效 convertBitmapToByteBuffer(位图位图){如果(imgData == null){返回;}imgData.rewind();bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());long startTime = SystemClock.uptimeMillis();//将图像转换为浮点数.整数像素 = 0;for (int i = 0; i < getImageSizeX(); ++i) {for (int j = 0; j < getImageSizeY(); ++j) {最终 int val = intValues[pixel++];imgData.putFloat(((val>>16) & 0xFF)/255.f);imgData.putFloat(((val>> 8) & 0xFF)/255.f);imgData.putFloat((val & 0xFF)/255.f);}}long endTime = SystemClock.uptimeMillis();Log.d(TAG, "将值放入 ByteBuffer 的时间成本:" + Long.toString(endTime - startTime));}

OpenCV 位图到 ByteBuffer :-

/** 将图像数据写入 {@code ByteBuffer}.*/私有无效 convertBitmapToByteBuffer(位图位图){如果(imgData == null){返回;}imgData.rewind();bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());long startTime = SystemClock.uptimeMillis();Mat bufmat = new Mat(197,197,CV_8UC3);Mat newmat = new Mat(197,197,CV_32FC3);Utils.bitmapToMat(bitmap,bufmat);Imgproc.cvtColor(bufmat,bufmat,Imgproc.COLOR_RGBA2RGB);列表<垫子>sp_im = new ArrayList(3);Core.split(bufmat,sp_im);sp_im.get(0).convertTo(sp_im.get(0),CV_32F,1.0/255/0);sp_im.get(1).convertTo(sp_im.get(1),CV_32F,1.0/255.0);sp_im.get(2).convertTo(sp_im.get(2),CV_32F,1.0/255.0);Core.merge(sp_im,newmat);//bufmat.convertTo(newmat,CV_32FC3,1.0/255.0);浮动 buf[] = 新浮动 [197*197*3];newmat.get(0,0,buf);//imgData.wrap(buf).order(ByteOrder.nativeOrder()).getFloat();imgData.order(ByteOrder.nativeOrder()).asFloatBuffer().put(buf);long endTime = SystemClock.uptimeMillis();Log.d(TAG, "将值放入 ByteBuffer 的时间成本:" + Long.toString(endTime - startTime));}

解决方案

1. 我认为您代码中的 255/0 是复制/粘贴错误，而不是真正的代码.
2. 我想知道纯 Java 解决方案的时间成本是多少，尤其是当您将其与推理的时间成本进行权衡时.对我来说，对于 Google mobilenet_v1_1.0_224 的位图稍大一点，简单的浮点缓冲区准备时间不到推理时间的 5%.
3. 我可以量化 tflite 模型(使用相同的 tflite_convert 实用程序从 .h5 生成 .tflite 文件.实际上可能有三个量化操作，但我只使用了两个:--inference_input_type=QUANTIZED_UINT8 和 --post_training_quantize.
   • 生成的模型大约是 float32 模型的 25%，这本身就是一项成就.
   • 生成的模型运行速度大约快两倍(至少在某些设备上).
   • 而且，生成的模型消耗 unit8 输入.这意味着代替 imgData.putFloat(((val>>16) & 0xFF)/255.f) 我们写 imgData.put((val>> 16) &; 0xFF)，依此类推.

顺便说一下，我不认为你的公式是正确的.为了在涉及 float32 缓冲区时获得最佳精度，我们使用

putFLoat(byteval/256f)

其中 byteval 是 [0:255] 范围内的整数.

In tensorflow-lite android demo code for image classification, the images are first converted to ByteBuffer format for better performance.This conversion from bitmap to floating point format and the subsequent conversion to byte buffer seems to be an expensive operation(loops, bitwise operators, float mem-copy etc).We were trying to implement the same logic with opencv to gain some speed advantage.The following code works without error; but due to some logical error in this conversion, the output of the model(to which this data is fed) seems to be incorrect.The input of the model is supposed to be RGB with data type float[1,197,197,3].

How can we speed up this process of bitmap to byte buffer conversion using opencv (or any other means)?

Standard Bitmap to ByteBuffer Conversion:-

/** Writes Image data into a {@code ByteBuffer}. */
  private void convertBitmapToByteBuffer(Bitmap bitmap) {
    if (imgData == null) {
      return;
    }
    imgData.rewind();


    bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());



    long startTime = SystemClock.uptimeMillis();

    // Convert the image to floating point.
    int pixel = 0;

    for (int i = 0; i < getImageSizeX(); ++i) {
      for (int j = 0; j < getImageSizeY(); ++j) {
        final int val = intValues[pixel++];

        imgData.putFloat(((val>> 16) & 0xFF) / 255.f);
        imgData.putFloat(((val>> 8) & 0xFF) / 255.f);
        imgData.putFloat((val & 0xFF) / 255.f);
      }
    }

    long endTime = SystemClock.uptimeMillis();
    Log.d(TAG, "Timecost to put values into ByteBuffer: " + Long.toString(endTime - startTime));
  }

OpenCV Bitmap to ByteBuffer :-

    /** Writes Image data into a {@code ByteBuffer}. */
      private void convertBitmapToByteBuffer(Bitmap bitmap) {
        if (imgData == null) {
          return;
        }
        imgData.rewind();


        bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());

        long startTime = SystemClock.uptimeMillis();


        Mat bufmat = new Mat(197,197,CV_8UC3);
        Mat newmat = new Mat(197,197,CV_32FC3);


        Utils.bitmapToMat(bitmap,bufmat);
        Imgproc.cvtColor(bufmat,bufmat,Imgproc.COLOR_RGBA2RGB);

        List<Mat> sp_im = new ArrayList<Mat>(3);


        Core.split(bufmat,sp_im);

        sp_im.get(0).convertTo(sp_im.get(0),CV_32F,1.0/255/0);
        sp_im.get(1).convertTo(sp_im.get(1),CV_32F,1.0/255.0);
        sp_im.get(2).convertTo(sp_im.get(2),CV_32F,1.0/255.0);

        Core.merge(sp_im,newmat);



        //bufmat.convertTo(newmat,CV_32FC3,1.0/255.0);
        float buf[] = new float[197*197*3];


        newmat.get(0,0,buf);

        //imgData.wrap(buf).order(ByteOrder.nativeOrder()).getFloat();
        imgData.order(ByteOrder.nativeOrder()).asFloatBuffer().put(buf);


        long endTime = SystemClock.uptimeMillis();
        Log.d(TAG, "Timecost to put values into ByteBuffer: " + Long.toString(endTime - startTime));
      }

解决方案

1. I believe that 255/0 in your code is a copy/paste mistake, not real code.
2. I wonder what the timecost of the pure Java solution is, especially when you weigh it against the timecost of inference. For me, with a slightly larger bitmap for Google's mobilenet_v1_1.0_224, the naïve float buffer preparation was less than 5% of inference time.
3. I could quantize the tflite model (with the same tflite_convert utility that generated .tflite file from .h5. There could actually be three quantization operations, but I only used two: --inference_input_type=QUANTIZED_UINT8 and --post_training_quantize.
   • The resulting model is about 25% size of the float32 one, which is an achievement on its own.
   • The resulting model runs about twice faster (at least on some devices).
   • And, the resulting model consumes unit8 inputs. This means that instead of imgData.putFloat(((val>> 16) & 0xFF) / 255.f) we write imgData.put((val>> 16) & 0xFF), and so on.

By the way, I don't think that your formulae are correct. To achieve best accuracy when float32 buffers are involved, we use

putFLoat(byteval / 256f)

where byteval is int in range [0:255].

这篇关于在 Tensorflow-lite Android 中将 Bitmap 转换为 ByteBuffer (float)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！