如何正确使用YUV_420_888和MediaCodec的ImageReader将视频编码为h264格式? [英] How to correctly use ImageReader with YUV_420_888 and MediaCodec to encode video to h264 format?

问题描述

我正在Android设备上实现相机应用程序.目前，我使用Camera2 API和ImageReader来获取YUV_420_888格式的图像数据，但是我不知道如何将这些数据准确地写入MediaCodec.

I'm implementing a camera application on Android devices. Currently, I use Camera2 API and ImageReader to get image data in YUV_420_888 format, but I don't know how to exactly write these data to MediaCodec.

这是我的问题:

1. 什么是YUV_420_888?

格式YUV_420_888模棱两可，因为它可以是属于YUV420系列的任何格式，例如YUV420P，YUV420PP，YUV420SP和YUV420PSP，对吗?

The format YUV_420_888 is ambiguous because it can be any format which belongs to the YUV420 family, such as YUV420P, YUV420PP, YUV420SP and YUV420PSP, right?

通过访问图像的三个平面(#0，#1，#2)，我可以获得该图像的Y(#0)，U(#1)，V(#2)值.但是这些值的排列在不同设备上可能不相同.例如，如果YUV_420_888确实表示YUV420P，则Plane#1和Plane#2的大小均为Plane#0的四分之一.如果YUV_420_888确实表示YUV420SP，则Plane#1和Plane#2的大小都是Plane#0大小的一半(Plane#1和Plane#2的每个包含U，V值).

By accessing the image's three planes(#0, #1, #2), I can get the Y(#0), U(#1), V(#2) values of this image. But the arrangement of these values may not be the same on different devices. For example, if YUV_420_888 truly means YUV420P, the size of both Plane#1 and Plane#2 is a quarter of the size of Plane#0. If YUV_420_888 truly means YUV420SP, the size of both Plane#1 and Plane#2 is half of the size of Plane#0(Each of Plane#1 and Plane#2 contains U, V values).

如果要将这些数据从图像的三个平面写入MediaCodec，我需要转换为哪种格式? YUV420，NV21，NV12，...?

If I want to write these data from image's three planes to MediaCodec, what kind of format I need to convert to? YUV420, NV21, NV12, ...?

1. 什么是COLOR_FormatYUV420Flexible?

格式COLOR_FormatYUV420Flexible也是模棱两可的，因为它可以是属于YUV420系列的任何格式，对吗?如果我将MediaCodec对象的KEY_COLOR_FORMAT选项设置为COLOR_FormatYUV420Flexible，我应该输入哪种格式的数据(YUV420P，YUV420SP ...?)?

The format COLOR_FormatYUV420Flexible is also ambiguous because it can be any format which belongs to the YUV420 family, right? If I set KEY_COLOR_FORMAT option of a MediaCodec object to COLOR_FormatYUV420Flexible, what format(YUV420P, YUV420SP...?) of data should I input to the MediaCodec object?

1. 如何使用COLOR_FormatSurface?

我知道MediaCodec有其自己的表面，如果我将MediaCodec对象的KEY_COLOR_FORMAT选项设置为COLOR_FormatSurface，则可以使用该表面.使用Camera2 API，我不需要自己将任何数据写入MediaCodec对象.我可以排空输出缓冲区.

I know MediaCodec has its own surface, which can be used if I set KEY_COLOR_FORMAT option of a MediaCodec object to COLOR_FormatSurface. And with Camera2 API, I don't need to write any data by myself to the MediaCodec object. I can just drain the output buffer.

但是，我需要更改相机的图像.例如，绘制其他图片，在上面写一些文字，或将另一个视频插入为POP(图片的图片).

However, I need to change the image from the camera. For example, draw other pictures, write some text on it, or insert another video as POP(Picture of Picture).

我可以使用ImageReader从Camera读取图像，然后重新绘制图像，然后将新数据写入MediaCodec的表面，然后将其排干吗?该怎么做?

Can I use ImageReader to read the image from Camera, and after re-drawing that, write the new data to MediaCodec's surface, and then drain it out? How to do that?

EDIT1

我通过使用COLOR_FormatSurface和RenderScript实现了该功能.这是我的代码:

I implemented the function by using COLOR_FormatSurface and RenderScript. Here is my code:

onImageAvailable方法:

public void onImageAvailable(ImageReader imageReader) {
    try {
        try (Image image = imageReader.acquireLatestImage()) {
            if (image == null) {
                return;
            }
            Image.Plane[] planes = image.getPlanes();
            if (planes.length >= 3) {
                ByteBuffer bufferY = planes[0].getBuffer();
                ByteBuffer bufferU = planes[1].getBuffer();
                ByteBuffer bufferV = planes[2].getBuffer();
                int lengthY = bufferY.remaining();
                int lengthU = bufferU.remaining();
                int lengthV = bufferV.remaining();
                byte[] dataYUV = new byte[lengthY + lengthU + lengthV];
                bufferY.get(dataYUV, 0, lengthY);
                bufferU.get(dataYUV, lengthY, lengthU);
                bufferV.get(dataYUV, lengthY + lengthU, lengthV);
                imageYUV = dataYUV;
            }
        }
    } catch (final Exception ex) {

    }
}

将YUV_420_888转换为RGB:

Convert YUV_420_888 to RGB:

public static Bitmap YUV_420_888_toRGBIntrinsics(Context context, int width, int height, byte[] yuv) {
    RenderScript rs = RenderScript.create(context);
    ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));

    Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs)).setX(yuv.length);
    Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);

    Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(width).setY(height);
    Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);


    Bitmap bmpOut = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);

    in.copyFromUnchecked(yuv);

    yuvToRgbIntrinsic.setInput(in);
    yuvToRgbIntrinsic.forEach(out);
    out.copyTo(bmpOut);
    return bmpOut;
}

MediaCodec:

MediaCodec:

mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
...
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
...
surface = mediaCodec.createInputSurface(); // This surface is not used in Camera APIv2. Camera APIv2 uses ImageReader's surface.

在其他线程中:

while (!stop) {
    final byte[] image = imageYUV;

    // Do some yuv computation

    Bitmap bitmap = YUV_420_888_toRGBIntrinsics(getApplicationContext(), width, height, image);
    Canvas canvas = surface.lockHardwareCanvas();
    canvas.drawBitmap(bitmap, matrix, paint);
    surface.unlockCanvasAndPost(canvas);
}

这种方式有效，但是性能不佳.它无法输出30fps的视频文件(仅〜12fps).也许我不应该使用COLOR_FormatSurface和表面的画布进行编码.计算得出的YUV数据应直接写入mediaCodec，而无需任何表面进行任何转换.但是我仍然不知道该怎么做.

This way works, but the performance is not good. It can't output 30fps video files(only ~12fps). Perhaps I should not use COLOR_FormatSurface and the surface's canvas for encoding. The computed YUV data should be written to the mediaCodec directly without any surface doing any conversion. But I still don't know how to do that.

推荐答案

是的，YUV_420_888是可以包装不同YUV 420格式的格式.该规范仔细地解释了U和V平面的布置没有规定，但是有一定的限制.例如如果U平面的像素跨度为2，则对V的像素跨度相同(然后基础字节缓冲区可以为NV21).

You are right, YUV_420_888 is a format that can wrap different YUV 420 formats. The spec carefully explains that the arrangement of U and V planes is not prescribed, but there are certain restrictions; e.g. if the U plane has pixel stride 2, same applies to V (and then the underlying byte buffer can be NV21).

COLOR_FormatYUV420Flexible 是 YUV_420_888 ，但它们分别属于不同的类:MediaCodec和ImageFormat

COLOR_FormatYUV420Flexible is a synonym of YUV_420_888, but they belong to different classes: MediaCodec and ImageFormat, respectively.

规范说明:

所有视频编解码器均支持灵活的YUV 4:2:0缓冲区，因为 COLOR_FormatSurface是一种不透明格式，可以为MediaCodec提供最佳性能，但这是有代价的:您不能直接读取或操纵其内容.如果您需要处理传给MediaCodec的数据，则可以使用ImageReader.它是否会比ByteBuffer效率更高，取决于您执行的操作和操作方式.请注意，对于 API 24+ ，您可以使用C ++中的camera2和MediaCodec.

COLOR_FormatSurface is an opaque format that can deliver best performance for MediaCodec, but this comes at price: you cannot directly read or manipulate its content. If you need to manipulate the data that goes to the MediaCodec, then using ImageReader is an option; whether it will be more efficient than ByteBuffer, depends on what you do and how you do it. Note that for API 24+ you can work with both camera2 and MediaCodec in C++.

MediaCodec的宝贵详细信息资源是 http://www.bigflake.com/mediacodec .它引用了264编码的完整示例.

The invaluable resource of details for MediaCodec is http://www.bigflake.com/mediacodec. It references a full example of 264 encoding.

这篇关于如何正确使用YUV_420_888和MediaCodec的ImageReader将视频编码为h264格式?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！