camera2捕获的图片-从YUV_420_888到NV21的转换 [英] camera2 captured picture - conversion from YUV_420_888 to NV21

问题描述

通过camera2 API，我们收到的图像对象的格式为 YUV_420_888 .然后，我们使用以下函数转换为 NV21 :

Via the camera2 API we are receiving an Image object of the format YUV_420_888. We are using then the following function for conversion to NV21:

private static byte[] YUV_420_888toNV21(Image image) {
    byte[] nv21;
    ByteBuffer yBuffer = image.getPlanes()[0].getBuffer();
    ByteBuffer uBuffer = image.getPlanes()[1].getBuffer();
    ByteBuffer vBuffer = image.getPlanes()[2].getBuffer();

    int ySize = yBuffer.remaining();
    int uSize = uBuffer.remaining();
    int vSize = vBuffer.remaining();

    nv21 = new byte[ySize + uSize + vSize];

    //U and V are swapped
    yBuffer.get(nv21, 0, ySize);
    vBuffer.get(nv21, ySize, vSize);
    uBuffer.get(nv21, ySize + vSize, uSize);

    return nv21;
}

虽然此函数在cameraCaptureSessions.setRepeatingRequest上可以正常工作，但在调用cameraCaptureSessions.capture时，在进一步处理中(在JNI端)会出现分段错误.两者都通过ImageReader要求YUV_420_888格式.

While this function works fine with cameraCaptureSessions.setRepeatingRequest, we get a segmentation error in further processing (on the JNI side) when calling cameraCaptureSessions.capture. Both request YUV_420_888 format via ImageReader.

在请求的类型相同的情况下，两个函数调用的结果为何不同?

How come the result is different for both function calls while the requested type is the same?

更新:如注释中所述，由于图像大小不同(捕获请求的尺寸大得多)，我得到了这种行为.但是我们在JNI方面的进一步处理操作对于这两个请求都是相同的，并且不依赖于图像尺寸(仅取决于纵横比，这在两种情况下都是相同的).

Update: As mentioned in the comments I get this behaviour because of different image sizes (much larger dimension for the capture request). But our further processing operations on the JNI side are the same for both requests and don't depend on image dimensions (only on the aspect ratio, which is in both cases the same).

推荐答案

如果根本没有填充，并且U和V平原重叠并且实际上表示隔行扫描的VU值，则您的代码将仅返回正确的NV21.这种情况在预览时经常发生，但是在这种情况下，您需要为数组分配额外的w * h/4字节(这大概不是问题).也许对于捕获的图像，您需要更强大的实现，例如

Your code will only return correct NV21 if there is no padding at all, and U and V plains overlap and actually represent interlaced VU values. This happens quite often for preview, but in such case you allocate extra w*h/4 bytes for your array (which presumably is not a problem). Maybe for captured image you need a more robust implemenation, e.g.

private static byte[] YUV_420_888toNV21(Image image) {

    int width = image.getWidth();
    int height = image.getHeight(); 
    int ySize = width*height;
    int uvSize = width*height/4;

    byte[] nv21 = new byte[ySize + uvSize*2];

    ByteBuffer yBuffer = image.getPlanes()[0].getBuffer(); // Y
    ByteBuffer uBuffer = image.getPlanes()[1].getBuffer(); // U
    ByteBuffer vBuffer = image.getPlanes()[2].getBuffer(); // V

    int rowStride = image.getPlanes()[0].getRowStride();
    assert(image.getPlanes()[0].getPixelStride() == 1);

    int pos = 0;

    if (rowStride == width) { // likely
        yBuffer.get(nv21, 0, ySize);
        pos += ySize;
    }
    else {
        long yBufferPos = -rowStride; // not an actual position
        for (; pos<ySize; pos+=width) {
            yBufferPos += rowStride;
            yBuffer.position(yBufferPos);
            yBuffer.get(nv21, pos, width);
        }
    }

    rowStride = image.getPlanes()[2].getRowStride();
    int pixelStride = image.getPlanes()[2].getPixelStride();

    assert(rowStride == image.getPlanes()[1].getRowStride());
    assert(pixelStride == image.getPlanes()[1].getPixelStride());
    
    if (pixelStride == 2 && rowStride == width && uBuffer.get(0) == vBuffer.get(1)) {
        // maybe V an U planes overlap as per NV21, which means vBuffer[1] is alias of uBuffer[0]
        byte savePixel = vBuffer.get(1);
        try {
            vBuffer.put(1, (byte)~savePixel);
        if (uBuffer.get(0) == (byte)~savePixel) {
            vBuffer.put(1, savePixel);
            vBuffer.get(nv21, ySize, uvSize);

            return nv21; // shortcut
        }
        catch (ReadOnlyBufferException ex) {
            // unfortunately, we cannot check if vBuffer and uBuffer overlap
        }

        // unfortunately, the check failed. We must save U and V pixel by pixel
        vBuffer.put(1, savePixel);
    }

    // other optimizations could check if (pixelStride == 1) or (pixelStride == 2), 
    // but performance gain would be less significant

    for (int row=0; row<height/2; row++) {
        for (int col=0; col<width/2; col++) {
            int vuPos = col*pixelStride + row*rowStride;
            nv21[pos++] = vBuffer.get(vuPos);
            nv21[pos++] = uBuffer.get(vuPos);
        }
    }

    return nv21;
}

如果您打算将结果数组传递给C ++，则可以利用事实

If you anyway intend to pass the resulting array to C++, you can take advantage of the fact that

返回的缓冲区将始终具有isDirect返回true，因此可以在JNI中将基础数据映射为指针，而无需使用GetDirectBufferAddress进行任何复制.

the buffer returned will always have isDirect return true, so the underlying data could be mapped as a pointer in JNI without doing any copies with GetDirectBufferAddress.

这意味着可以使用最少的开销在C ++中完成相同的转换.在C ++中，您甚至可能发现实际的像素排列已经是NV21！

This means that same conversion may be done in C++ with minimal overhead. In C++, you may even find that the actual pixel arrangement is already NV21!

PS 实际上，这可以用Java完成，而开销却可以忽略不计，请参见上面的行if (pixelStride == 2 && ….因此，我们可以将所有色度字节批量复制到结果字节数组中，这比运行循环快得多，但仍比在C ++中实现这种情况要慢.

PS Actually, this can be done in Java, with negligible overhead, see the line if (pixelStride == 2 && … above. So, we can bulk copy all chroma bytes to the resulting byte array, which is much faster than running the loops, but still slower than what can be achieved for such case in C++.

这篇关于camera2捕获的图片-从YUV_420_888到NV21的转换的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！