如何正确使用带有 YUV_420_888 和 MediaCodec 的 ImageReader 将视频编码为 h264 格式? [英] How to correctly use ImageReader with YUV_420_888 and MediaCodec to encode video to h264 format?
问题描述
我正在 Android 设备上实现一个相机应用程序.目前,我使用Camera2 API和ImageReader获取YUV_420_888
格式的图像数据,但我不知道如何将这些数据准确写入MediaCodec.
I'm implementing a camera application on Android devices. Currently, I use Camera2 API and ImageReader to get image data in YUV_420_888
format, but I don't know how to exactly write these data to MediaCodec.
这是我的问题:
- 什么是
YUV_420_888
?
格式YUV_420_888
是不明确的,因为它可以是属于YUV420
家族的任何格式,例如YUV420P
、YUV420PP
、YUV420SP
和 YUV420PSP
对吧?
The format YUV_420_888
is ambiguous because it can be any format which belongs to the YUV420
family, such as YUV420P
, YUV420PP
, YUV420SP
and YUV420PSP
, right?
通过访问图像的三个平面(#0、#1、#2),我可以获得该图像的 Y(#0)、U(#1)、V(#2) 值.但是这些值的排列在不同的设备上可能不一样.例如,如果YUV_420_888
真正表示YUV420P
,则Plane#1和Plane#2的大小都是Plane#0大小的四分之一.如果YUV_420_888
真的是YUV420SP
,那么Plane#1和Plane#2的大小都是Plane#0大小的一半(Plane#1和Plane#2各有一个包含 U、V 值).
By accessing the image's three planes(#0, #1, #2), I can get the Y(#0), U(#1), V(#2) values of this image. But the arrangement of these values may not be the same on different devices. For example, if YUV_420_888
truly means YUV420P
, the size of both Plane#1 and Plane#2 is a quarter of the size of Plane#0. If YUV_420_888
truly means YUV420SP
, the size of both Plane#1 and Plane#2 is half of the size of Plane#0(Each of Plane#1 and Plane#2 contains U, V values).
如果我想把图像三个平面的这些数据写入MediaCodec,需要转换成什么样的格式?YUV420, NV21, NV12, ...?
If I want to write these data from image's three planes to MediaCodec, what kind of format I need to convert to? YUV420, NV21, NV12, ...?
- 什么是
COLOR_FormatYUV420Flexible
?
格式COLOR_FormatYUV420Flexible
也有歧义,因为它可以是属于YUV420
家族的任何格式,对吧?如果我将 MediaCodec 对象的 KEY_COLOR_FORMAT
选项设置为 COLOR_FormatYUV420Flexible
,我应该向 MediaCodec 对象输入什么格式的数据(YUV420P、YUV420SP...?)?
The format COLOR_FormatYUV420Flexible
is also ambiguous because it can be any format which belongs to the YUV420
family, right? If I set KEY_COLOR_FORMAT
option of a MediaCodec object to COLOR_FormatYUV420Flexible
, what format(YUV420P, YUV420SP...?) of data should I input to the MediaCodec object?
- 如何使用
COLOR_FormatSurface
?
我知道 MediaCodec 有自己的表面,如果我将 MediaCodec 对象的 KEY_COLOR_FORMAT
选项设置为 COLOR_FormatSurface
,就可以使用它.并且使用 Camera2 API,我不需要自己向 MediaCodec 对象写入任何数据.我可以排空输出缓冲区.
I know MediaCodec has its own surface, which can be used if I set KEY_COLOR_FORMAT
option of a MediaCodec object to COLOR_FormatSurface
. And with Camera2 API, I don't need to write any data by myself to the MediaCodec object. I can just drain the output buffer.
但是,我需要更改相机中的图像.例如,绘制其他图片,在上面写一些文字,或者插入另一个视频作为 POP(图片图片).
However, I need to change the image from the camera. For example, draw other pictures, write some text on it, or insert another video as POP(Picture of Picture).
我可以使用ImageReader从Camera读取图像,重新绘制后,将新数据写入MediaCodec的表面,然后将其排出吗?怎么做?
Can I use ImageReader to read the image from Camera, and after re-drawing that, write the new data to MediaCodec's surface, and then drain it out? How to do that?
EDIT1
我使用COLOR_FormatSurface
和RenderScript 实现了该功能.这是我的代码:
I implemented the function by using COLOR_FormatSurface
and RenderScript. Here is my code:
onImageAvailable
方法:
public void onImageAvailable(ImageReader imageReader) {
try {
try (Image image = imageReader.acquireLatestImage()) {
if (image == null) {
return;
}
Image.Plane[] planes = image.getPlanes();
if (planes.length >= 3) {
ByteBuffer bufferY = planes[0].getBuffer();
ByteBuffer bufferU = planes[1].getBuffer();
ByteBuffer bufferV = planes[2].getBuffer();
int lengthY = bufferY.remaining();
int lengthU = bufferU.remaining();
int lengthV = bufferV.remaining();
byte[] dataYUV = new byte[lengthY + lengthU + lengthV];
bufferY.get(dataYUV, 0, lengthY);
bufferU.get(dataYUV, lengthY, lengthU);
bufferV.get(dataYUV, lengthY + lengthU, lengthV);
imageYUV = dataYUV;
}
}
} catch (final Exception ex) {
}
}
将 YUV_420_888 转换为 RGB:
Convert YUV_420_888 to RGB:
public static Bitmap YUV_420_888_toRGBIntrinsics(Context context, int width, int height, byte[] yuv) {
RenderScript rs = RenderScript.create(context);
ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));
Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs)).setX(yuv.length);
Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);
Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(width).setY(height);
Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);
Bitmap bmpOut = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
in.copyFromUnchecked(yuv);
yuvToRgbIntrinsic.setInput(in);
yuvToRgbIntrinsic.forEach(out);
out.copyTo(bmpOut);
return bmpOut;
}
媒体编解码器:
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
...
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
...
surface = mediaCodec.createInputSurface(); // This surface is not used in Camera APIv2. Camera APIv2 uses ImageReader's surface.
在另一个线程中:
while (!stop) {
final byte[] image = imageYUV;
// Do some yuv computation
Bitmap bitmap = YUV_420_888_toRGBIntrinsics(getApplicationContext(), width, height, image);
Canvas canvas = surface.lockHardwareCanvas();
canvas.drawBitmap(bitmap, matrix, paint);
surface.unlockCanvasAndPost(canvas);
}
这种方式有效,但性能不佳.它不能输出 30fps 的视频文件(只有 ~12fps).也许我不应该使用 COLOR_FormatSurface
和表面的画布进行编码.计算出的 YUV 数据应直接写入 mediaCodec,无需任何表面进行任何转换.但我还是不知道该怎么做.
This way works, but the performance is not good. It can't output 30fps video files(only ~12fps). Perhaps I should not use COLOR_FormatSurface
and the surface's canvas for encoding. The computed YUV data should be written to the mediaCodec directly without any surface doing any conversion. But I still don't know how to do that.
推荐答案
你说得对,YUV_420_888 是一种可以包装不同 YUV 420 格式的格式.规范仔细解释了U和V平面的排列没有规定,但有一定的限制;例如如果 U 平面的像素步长为 2,则同样适用于 V(然后底层字节缓冲区可以是 NV21).
You are right, YUV_420_888 is a format that can wrap different YUV 420 formats. The spec carefully explains that the arrangement of U and V planes is not prescribed, but there are certain restrictions; e.g. if the U plane has pixel stride 2, same applies to V (and then the underlying byte buffer can be NV21).
COLOR_FormatYUV420Flexible 是
COLOR_FormatYUV420Flexible is a synonym of YUV_420_888, but they belong to different classes: MediaCodec and ImageFormat, respectively.
规范解释:
所有视频编解码器都支持灵活的 YUV 4:2:0 缓冲区,因为 LOLLIPOP_MR1.
All video codecs support flexible YUV 4:2:0 buffers since LOLLIPOP_MR1.
COLOR_FormatSurface 是一种不透明格式,可为 MediaCodec 提供最佳性能,但这是有代价的:您无法直接读取或操作其内容.如果您需要操作进入 MediaCodec 的数据,那么使用 ImageReader 是一种选择;它是否会比 ByteBuffer 更有效,取决于你做什么以及你如何做.请注意,对于 API 24+,您可以使用C++中的camera2和MediaCodec.
COLOR_FormatSurface is an opaque format that can deliver best performance for MediaCodec, but this comes at price: you cannot directly read or manipulate its content. If you need to manipulate the data that goes to the MediaCodec, then using ImageReader is an option; whether it will be more efficient than ByteBuffer, depends on what you do and how you do it. Note that for API 24+ you can work with both camera2 and MediaCodec in C++.
MediaCodec 的宝贵细节资源是 http://www.bigflake.com/mediacodec.它引用了 264 编码的完整示例.
The invaluable resource of details for MediaCodec is http://www.bigflake.com/mediacodec. It references a full example of 264 encoding.
这篇关于如何正确使用带有 YUV_420_888 和 MediaCodec 的 ImageReader 将视频编码为 h264 格式?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!