Xamarin tf.lite输入对象 [英] Xamarin tf.lite input objects

问题描述

我正在尝试在xamarin上重现Tensorflow对象检测.

Im trying to reproduce tensorflow object detection on xamarin.

    private MappedByteBuffer LoadModelFile()
    {
        AssetFileDescriptor fileDescriptor = Assets.OpenFd("detect.tflite");           
        FileInputStream inputStream = new FileInputStream(fileDescriptor.FileDescriptor);
        FileChannel fileChannel = inputStream.Channel;
        long startOffset = fileDescriptor.StartOffset;
        long declaredLength = fileDescriptor.DeclaredLength;
        return fileChannel.Map(FileChannel.MapMode.ReadOnly, startOffset, declaredLength);
    }

        View view = (View) sender;

        MappedByteBuffer buffer = LoadModelFile();

        Interpreter interpreter = new Interpreter(buffer);

        var sr = new StreamReader(Assets.Open("labels.txt"));

        var labels = sr.ReadToEnd()
            .Split('\n')
            .Select(s => s.Trim())
            .Where(s => !string.IsNullOrEmpty(s))
            .ToList();

        var bitmap =  BitmapFactory.DecodeResource(Resources, 2130837608);
        var resizedBitmap = Bitmap.CreateScaledBitmap(bitmap, 1000, 750, false)
            .Copy(Bitmap.Config.Argb8888, false);

        float[][][][] imgData = null;

        imgData  = new float[1][][][];
        imgData[0] = new float[1000][][];

        for (int i = 0; i < imgData[0].Length; i++)
        {
            imgData[0][i] = new float[750][];
            for (int j = 0; j < imgData[0][i].Length; j++)
            {
                imgData[0][i][j] = new float[3];
            }
        }
        var intValuess = new int[1000 * 750];
        resizedBitmap.GetPixels(intValuess, 0, 1000, 0, 0, 1000, 750);

        int pixels = 0;
        for (int i = 0; i < imgData[0].Length; i++)
        {
            for (int j = 0; j < imgData[0][i].Length; j++)
            {
                var val = intValuess[pixels++];
                imgData[0][i][j][0] = (float)((val >> 16) & 0xFF);
                imgData[0][i][j][1] = (float)((val >> 8) & 0xFF);
                imgData[0][i][j][2] = (float)(val & 0xFF);
            }
        }

        var outputs = new float[labels.Count];

        interpreter.Run(imgData, outputs);

但是我有错误无法将float [] [] [] [] []转换为行解释器中的Java.Lang.Object. 我如何将float [] [] [] []转换为Java.Lang.Object或在哪里可以找到带有xamarin示例的tensorflow lite.

but i have error "cannot convert float[][][][] to Java.Lang.Object in line interpreter.Run(imgData, outputs); How i can convert float[][][][] to Java.Lang.Object or where i can find tensorflow lite with xamarin examples.

推荐答案

我知道距您提出此问题已有一段时间了，但也许我的回答对某人有用. 我还尝试将Xamarin与tflite一起使用，以运行简单的CNN. 这是我的代码:

I know it has been a while since you asked this question but maybe my response can be useful to someone. I am also trying to use Xamarin with tflite, to run a simple CNN. Here is my code:

private MappedByteBuffer LoadModelFile()
{
    var assets = Application.Context.Assets;
    AssetFileDescriptor fileDescriptor = assets.OpenFd("seed_model_no_qt.tflite");
    FileInputStream inputStream = new FileInputStream(fileDescriptor.FileDescriptor);
    FileChannel fileChannel = inputStream.Channel;
    long startOffset = fileDescriptor.StartOffset;
    long declaredLength = fileDescriptor.DeclaredLength;
    return fileChannel.Map(FileChannel.MapMode.ReadOnly, startOffset, declaredLength);
}

private string Classify(MediaFile mediaFile)
{
    var assets = Application.Context.Assets;

    Bitmap bp = BitmapFactory.DecodeStream(mediaFile.GetStream());
    var resizedBitmap = Bitmap.CreateScaledBitmap(bp, 1280, 1280, false).Copy(Bitmap.Config.Argb8888, false);

    var bufint = new int[1280 * 1280];
    resizedBitmap.GetPixels(bufint, 0, 1280, 0, 0, 1280, 1280);
    int pixels = 0;
    var input_buffer = new byte[4 * 1280 * 1280 * 3];            
    for(int i = 0; i < 1280; i++)
    {
        for(int k = 0; k < 1280; k++)
        {
            int val = bufint[pixels++];
            Array.Copy(BitConverter.GetBytes(((val >> 16) & 0xFF) * (1f / 255f)), 0, input_buffer, (i * 1280 + k) * 12, 4);
            Array.Copy(BitConverter.GetBytes(((val >> 8) & 0xFF) * (1f / 255f)), 0, input_buffer, (i * 1280 + k) * 12 + 4, 4);
            Array.Copy(BitConverter.GetBytes((val & 0xFF) * (1f / 255f)), 0, input_buffer, (i * 1280 + k) * 12 + 8, 4);
        }
    }
    var bytebuffer = Java.Nio.ByteBuffer.Wrap(input_buffer);
    var output = Java.Nio.ByteBuffer.AllocateDirect(4*160*160);
    interpreter.Run(bytebuffer, output);

    var buffer = new byte[4 * 160 * 160];

    Marshal.Copy(output.GetDirectBufferAddress(), buffer, 0, 4 * 160 * 160);

    float sum = 0.0f;
    for(int i = 0; i < 160*160; i++)
    {
        sum += BitConverter.ToSingle(buffer, i * 4);
    }

    return "Count : " + ((int)(sum/255)).ToString();
}

我按原样重用了您的LoadModelFile()函数.该代码从mediaFile(来自电话摄像头)获取图像，然后将其大小调整为1280x1280 rgb图像，然后再将其作为float32值数组提供给CNN. 您对Java.Lang.Object的float [] [] [] []问题来自期望Java对象的interpreter.Run()方法.有人在线通过提供Java.Nio.ByteBuffer作为参数而不是数组来解决它.它暗示了一些按位操作，但Run方法确实接受ByteBuffer对象. 填充ByteBuffer时，建议您不要使用PutFloat()之类的方法，而是填充byte []缓冲区，然后像我一样使用Java.Nio.ByteBuffer.Wrap()方法.在我的情况下，使用ByteBuffer的方法似乎暗示了较大的性能问题. 操纵CNN的输出(float32值的160x160热图)时，也会发生相同的情况.使用ByteBuffer.Get()方法访问值非常慢.而是使用Marshal.Copy将值存储到字节数组中，然后使用BitConverter.ToSingle获取浮点值.

I reused your LoadModelFile() function as it is. The code takes an image from a mediaFile (coming from the phone camera), then resizes it to 1280x1280 rgb image before feeding it to a CNN as an array of float32 values. Your float[][][][] to Java.Lang.Object issue came from the interpreter.Run() method expecting a Java Object. Some people online solve it by giving a Java.Nio.ByteBuffer as a parameter, instead of an array. It implies some bitwise manipulations but the Run method does accept the ByteBuffer object. When filling the ByteBuffer, I advise you not to use its methods such as PutFloat(), but to fill a byte[] buffer and then use the Java.Nio.ByteBuffer.Wrap() method as I did. Using ByteBuffer's methods seemed to imply large performance issues in my case. Same thing happens when manipulating the output of my CNN (a 160x160 heatmap of float32 values). Using ByteBuffer.Get() method to access the values was very slow. Instead, use Marshal.Copy to store the values into a byte array, then get back the float values with BitConverter.ToSingle.

这篇关于Xamarin tf.lite输入对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！