如何转换 &在 onImageAvailable (android Camera2) 中从前置摄像头纵向模式旋转原始 NV21 阵列图像 (android.media.Image)? [英] How to convert & rotate raw NV21 array image (android.media.Image) from front cam portrait mode in onImageAvailable (android Camera2)?

问题描述

注意:我帖子中的所有信息仅适用于三星 Galaxy S7 设备.我不知道模拟器和其他设备的行为.

在 onImageAvailable 中，我连续将每个图像转换为 NV21 字节数组，并将其转发到需要原始 NV21 格式的 API.

这是我初始化图像阅读器并接收图像的方式:

private void openCamera() {...mImageReader = ImageReader.newInstance(WIDTH, HEIGHT,ImageFormat.YUV_420_888, 1);//只有 1 以获得最佳性能mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);...}私有最终 ImageReader.OnImageAvailableListener mOnImageAvailableListener= 新 ImageReader.OnImageAvailableListener() {@覆盖public void onImageAvailable(ImageReader reader) {Image image = reader.acquireLatestImage();如果(图像！= null){字节 [] 数据 = convertYUV420ToNV21_ALL_PLANES(image);//此图像在纵向模式下使用前置摄像头旋转 90 度字节[] data_rotated = rotateNV21_working(数据，宽度，高度，270)；ForwardToAPI(data_rotated);//图像数据被转发到 api 并稍后接收图像.关闭()；}}};

将图像转换为原始 NV21 的函数(

注意:它仍然是同一个杯子，只是你看到了 3-4 次.

使用另一个建议的旋转函数

上图显示了使用纹理视图表面并将其添加到 captureRequestBuilder 的直接流.下图显示了旋转后的原始图像数据.

问题是:

• convertYUV420ToNV21_ALL_PLANES"中的这个 hack 是否适用于任何设备/模拟器?
• 为什么rotateNV21 不起作用，而rotateNV21_working 工作正常.

镜像问题已修复，请参阅代码注释.挤压问题已修复，这是由它转发的 API 引起的.实际开放的问题是一个适当的不太昂贵的功能，将图像转换和旋转为可在任何设备上运行的原始 NV21.

解决方案

这里是将图像转换为NV21字节[]的代码.当 imgYUV420 U 和 V 平面的 pixelStride=1(在模拟器上)或 pixelStride=2(在 Nexus 上)时，这将起作用:

private byte[] convertYUV420ToNV21_ALL_PLANES(Image imgYUV420) {断言(imgYUV420.getFormat() == ImageFormat.YUV_420_888);Log.d(TAG, "图像: " + imgYUV420.getWidth() + "x" + imgYUV420.getHeight() + " " + imgYUV420.getFormat());Log.d(TAG, "飞机:" + imgYUV420.getPlanes().length);for (int nplane = 0; nplane < imgYUV420.getPlanes().length; nplane++) {Log.d(TAG, "plane[" + nplane + "]: length " + imgYUV420.getPlanes()[nplane].getBuffer().remaining() + ", strides: " + imgYUV420.getPlanes()[nplane].getPixelStride() + " " + imgYUV420.getPlanes()[nplane].getRowStride());}字节[] rez = 新字节[imgYUV420.getWidth() * imgYUV420.getHeight() * 3/2];ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();ByteBuffer buffer1 = imgYUV420.getPlanes()[1].getBuffer();ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();整数 n = 0;断言(imgYUV420.getPlanes()[0].getPixelStride() == 1);for (int row = 0; row < imgYUV420.getHeight(); row++) {for (int col = 0; col < imgYUV420.getWidth(); col++) {rez[n++] = buffer0.get();}}断言(imgYUV420.getPlanes()[2].getPixelStride() == imgYUV420.getPlanes()[1].getPixelStride());int stride = imgYUV420.getPlanes()[1].getPixelStride();for (int row = 0; row < imgYUV420.getHeight(); row += 2) {for (int col = 0; col  0){buffer1.get();}如果(buffer2.remaining()> 0){buffer2.get();}}}}Log.w(TAG, "total: " + rez.length);返回 rez;}

优化的 Java 代码可在 以查看原始图像的实际结构.

Note: All info in my post only goes for Samsung Galaxy S7 device. I do not know how emulators and other devices behave.

In onImageAvailable I convert continuously each image to a NV21 byte array and forward it to an API expecting raw NV21 format.

This is how I initialize the image reader and receive the images:

private void openCamera() {
    ...
    mImageReader = ImageReader.newInstance(WIDTH, HEIGHT,
            ImageFormat.YUV_420_888, 1); // only 1 for best performance
    mImageReader.setOnImageAvailableListener(
    mOnImageAvailableListener, mBackgroundHandler);
    ...
}

private final ImageReader.OnImageAvailableListener mOnImageAvailableListener
        = new ImageReader.OnImageAvailableListener() {

    @Override
    public void onImageAvailable(ImageReader reader) {
        Image image = reader.acquireLatestImage();
        if (image != null) {
            byte[] data = convertYUV420ToNV21_ALL_PLANES(image); // this image is turned 90 deg using front cam in portrait mode
            byte[] data_rotated = rotateNV21_working(data, WIDTH, HEIGHT, 270);
            ForwardToAPI(data_rotated); // image data is being forwarded to api and received later on
            image.close();
        }
    }
};

The function converting the image to raw NV21 (from here), working fine, the image is (due to android?) turned by 90 degrees when using front cam in portrait mode: (I modified it, slightly according to comments of Alex Cohn)

private byte[] convertYUV420ToNV21_ALL_PLANES(Image imgYUV420) {

    byte[] rez;

    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();
    ByteBuffer buffer1 = imgYUV420.getPlanes()[1].getBuffer();
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();

    // actually here should be something like each second byte
    // however I simply get the last byte of buffer 2 and the entire buffer 1
    int buffer0_size = buffer0.remaining();
    int buffer1_size = buffer1.remaining(); // / 2 + 1;
    int buffer2_size = 1;//buffer2.remaining(); // / 2 + 1;

    byte[] buffer0_byte = new byte[buffer0_size];
    byte[] buffer1_byte = new byte[buffer1_size];
    byte[] buffer2_byte = new byte[buffer2_size];

    buffer0.get(buffer0_byte, 0, buffer0_size);
    buffer1.get(buffer1_byte, 0, buffer1_size);
    buffer2.get(buffer2_byte, buffer2_size-1, buffer2_size);


    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    try {
        // swap 1 and 2 as blue and red colors are swapped
        outputStream.write(buffer0_byte);
        outputStream.write(buffer2_byte);
        outputStream.write(buffer1_byte);
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    rez = outputStream.toByteArray();

    return rez;
}

Hence "data" needs to be rotated. Using this function (from here), I get a weird 3-times interlaced picture error:

public static byte[] rotateNV21(byte[] input, int width, int height, int rotation) {
    byte[] output = new byte[input.length];
    boolean swap = (rotation == 90 || rotation == 270);
    // **EDIT:** in portrait mode & front cam this needs to be set to true:
    boolean yflip = true;// (rotation == 90 || rotation == 180);
    boolean xflip = (rotation == 270 || rotation == 180);
    for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
            int xo = x, yo = y;
            int w = width, h = height;
            int xi = xo, yi = yo;
            if (swap) {
                xi = w * yo / h;
                yi = h * xo / w;
            }
            if (yflip) {
                yi = h - yi - 1;
            }
            if (xflip) {
                xi = w - xi - 1;
            }
            output[w * yo + xo] = input[w * yi + xi];
            int fs = w * h;
            int qs = (fs >> 2);
            xi = (xi >> 1);
            yi = (yi >> 1);
            xo = (xo >> 1);
            yo = (yo >> 1);
            w = (w >> 1);
            h = (h >> 1);
            // adjust for interleave here
            int ui = fs + (w * yi + xi) * 2;
            int uo = fs + (w * yo + xo) * 2;
            // and here
            int vi = ui + 1;
            int vo = uo + 1;
            output[uo] = input[ui];
            output[vo] = input[vi];
        }
    }
    return output;
}

Resulting into this picture:

Note: it is still the same cup, however you see it 3-4 times.

Using another suggested rotate function from here gives the proper result:

public static byte[] rotateNV21_working(final byte[] yuv,
                                final int width,
                                final int height,
                                final int rotation)
{
  if (rotation == 0) return yuv;
  if (rotation % 90 != 0 || rotation < 0 || rotation > 270) {
    throw new IllegalArgumentException("0 <= rotation < 360, rotation % 90 == 0");
  }

  final byte[]  output    = new byte[yuv.length];
  final int     frameSize = width * height;
  final boolean swap      = rotation % 180 != 0;
  final boolean xflip     = rotation % 270 != 0;
  final boolean yflip     = rotation >= 180;

  for (int j = 0; j < height; j++) {
    for (int i = 0; i < width; i++) {
      final int yIn = j * width + i;
      final int uIn = frameSize + (j >> 1) * width + (i & ~1);
      final int vIn = uIn       + 1;

      final int wOut     = swap  ? height              : width;
      final int hOut     = swap  ? width               : height;
      final int iSwapped = swap  ? j                   : i;
      final int jSwapped = swap  ? i                   : j;
      final int iOut     = xflip ? wOut - iSwapped - 1 : iSwapped;
      final int jOut     = yflip ? hOut - jSwapped - 1 : jSwapped;

      final int yOut = jOut * wOut + iOut;
      final int uOut = frameSize + (jOut >> 1) * wOut + (iOut & ~1);
      final int vOut = uOut + 1;

      output[yOut] = (byte)(0xff & yuv[yIn]);
      output[uOut] = (byte)(0xff & yuv[uIn]);
      output[vOut] = (byte)(0xff & yuv[vIn]);
    }
  }
  return output;
}

The result is fine now:

The top image shows the direct stream using a texture view's surface and adding it to the captureRequestBuilder. The bottom image shows the raw image data after rotating.

The questions are:

• Does this hack in "convertYUV420ToNV21_ALL_PLANES" work on any device/emulator?
• Why does rotateNV21 not work, while rotateNV21_working works fine.

Edit: The mirror issue is fixed, see code comment. The squeeze issue is fixed, it was caused by the API it gets forwarded. The actual open issue is a proper not too expensive function, converting and rotating an image into raw NV21 working on any device.

解决方案

Here is the code to convert the Image to NV21 byte[]. This will work when the imgYUV420 U and V planes have pixelStride=1 (as on emulator) or pixelStride=2 (as on Nexus):

private byte[] convertYUV420ToNV21_ALL_PLANES(Image imgYUV420) {

    assert(imgYUV420.getFormat() == ImageFormat.YUV_420_888);
    Log.d(TAG, "image: " + imgYUV420.getWidth() + "x" + imgYUV420.getHeight() + " " + imgYUV420.getFormat());
    Log.d(TAG, "planes: " + imgYUV420.getPlanes().length);
    for (int nplane = 0; nplane < imgYUV420.getPlanes().length; nplane++) {
        Log.d(TAG, "plane[" + nplane + "]: length " + imgYUV420.getPlanes()[nplane].getBuffer().remaining() + ", strides: " + imgYUV420.getPlanes()[nplane].getPixelStride() + " " + imgYUV420.getPlanes()[nplane].getRowStride());
    }

    byte[] rez = new byte[imgYUV420.getWidth() * imgYUV420.getHeight() * 3 / 2];
    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();
    ByteBuffer buffer1 = imgYUV420.getPlanes()[1].getBuffer();
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();

    int n = 0;
    assert(imgYUV420.getPlanes()[0].getPixelStride() == 1);
    for (int row = 0; row < imgYUV420.getHeight(); row++) {
        for (int col = 0; col < imgYUV420.getWidth(); col++) {
            rez[n++] = buffer0.get();
        }
    }
    assert(imgYUV420.getPlanes()[2].getPixelStride() == imgYUV420.getPlanes()[1].getPixelStride());
    int stride = imgYUV420.getPlanes()[1].getPixelStride();
    for (int row = 0; row < imgYUV420.getHeight(); row += 2) {
        for (int col = 0; col < imgYUV420.getWidth(); col += 2) {
            rez[n++] = buffer1.get();
            rez[n++] = buffer2.get();
            for (int skip = 1; skip < stride; skip++) {
                if (buffer1.remaining() > 0) {
                    buffer1.get();
                }
                if (buffer2.remaining() > 0) {
                    buffer2.get();
                }
            }
        }
    }

    Log.w(TAG, "total: " + rez.length);
    return rez;
}

optimized Java code is available here.

As you can see, it is very easy to change this code to produce a rotated image in a single step:

private byte[] rotateYUV420ToNV21(Image imgYUV420) {

    Log.d(TAG, "image: " + imgYUV420.getWidth() + "x" + imgYUV420.getHeight() + " " + imgYUV420.getFormat());
    Log.d(TAG, "planes: " + imgYUV420.getPlanes().length);
    for (int nplane = 0; nplane < imgYUV420.getPlanes().length; nplane++) {
        Log.d(TAG, "plane[" + nplane + "]: length " + imgYUV420.getPlanes()[nplane].getBuffer().remaining() + ", strides: " + imgYUV420.getPlanes()[nplane].getPixelStride() + " " + imgYUV420.getPlanes()[nplane].getRowStride());
    }

    byte[] rez = new byte[imgYUV420.getWidth() * imgYUV420.getHeight() * 3 / 2];
    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();
    ByteBuffer buffer1 = imgYUV420.getPlanes()[1].getBuffer();
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();

    int width = imgYUV420.getHeight();
    assert(imgYUV420.getPlanes()[0].getPixelStride() == 1);
    for (int row = imgYUV420.getHeight()-1; row >=0; row--) {
        for (int col = 0; col < imgYUV420.getWidth(); col++) {
            rez[col*width+row] = buffer0.get();
        }
    }
    int uv_offset = imgYUV420.getWidth()*imgYUV420.getHeight();
    assert(imgYUV420.getPlanes()[2].getPixelStride() == imgYUV420.getPlanes()[1].getPixelStride());
    int stride = imgYUV420.getPlanes()[1].getPixelStride();
    for (int row = imgYUV420.getHeight() - 2; row >= 0; row -= 2) {
        for (int col = 0; col < imgYUV420.getWidth(); col += 2) {
            rez[uv_offset+col/2*width+row] = buffer1.get();
            rez[uv_offset+col/2*width+row+1] = buffer2.get();
            for (int skip = 1; skip < stride; skip++) {
                if (buffer1.remaining() > 0) {
                    buffer1.get();
                }
                if (buffer2.remaining() > 0) {
                    buffer2.get();
                }
            }
        }
    }

    Log.w(TAG, "total rotated: " + rez.length);
    return rez;
}

I sincerely recommend the site http://rawpixels.net/ to see the actual structure of your raw images.

这篇关于如何转换 &在 onImageAvailable (android Camera2) 中从前置摄像头纵向模式旋转原始 NV21 阵列图像 (android.media.Image)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！