MediaCodec 和相机:色彩空间不匹配 [英] MediaCodec and Camera: colorspaces don't match
问题描述
我一直在尝试使用新的低级 MediaCodec.我在这方面遇到了一些困难，因为 MediaCodecAPI 的文档很差，但我终于找到了一些工作.
I have been trying to get H264 encoding to work with input captured by the camera on an Android tablet using the new low-level MediaCodec. I have gone through some difficulties with this, since the MediaCodecAPI is poorly documented, but I've gotten something to work at last.
我正在按如下方式设置相机:
I'm setting up the camera as follows:
Camera.Parameters parameters = mCamera.getParameters();
parameters.setPreviewFormat(ImageFormat.YV12); // <1>
parameters.setPreviewFpsRange(4000,60000);
parameters.setPreviewSize(640, 480);
mCamera.setParameters(parameters);
对于编码部分，我将 MediaCodec 对象实例化如下:
For the encoding part, I'm instantiating the MediaCodec object as follows:
mediaCodec = MediaCodec.createEncoderByType("video/avc");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 640, 480);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 500000);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 15);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT,
MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar); // <2>
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();
最终目标是创建 RTP 流(并与 Skype 对应)，但到目前为止我只是将原始 H264 直接流式传输到我的桌面.在那里我使用以下 GStreamer-pipeline 来显示结果:
The final goal is to create an RTP-stream (and correspond with Skype), but so far I am only streaming the raw H264 directly to my desktop. There I use the following GStreamer-pipeline to show the result:
gst-launch udpsrc port=5555 ! video/x-h264,width=640,height=480,framerate=15/1 ! ffdec_h264 ! autovideosink
一切都很好，除了颜色.我需要在计算机中设置 2 种颜色格式:一种用于相机预览(标记为 <1> 的行)和一种用于 MediaCodec 对象(标记为 <2>)
All works well, except for the colors. I need to set 2 colorformats in the computer: one for the camera-preview (line tagged with <1>) and one for the MediaCodec-object (tagged with <2>)
为了确定行 <1> 的可接受值，我使用了 parameters.getSupportedPreviewFormats().由此，我知道相机上唯一支持的格式是 ImageFormat.NV21 和 ImageFormat.YV2.
To determine the acceptable values for the lines <1> I used parameters.getSupportedPreviewFormats(). From this, I know that the only supported formats on the camera are ImageFormat.NV21 and ImageFormat.YV2.
对于 <2>，我检索了 MediaCodecInfo.CodecCapabilities-video/avc 类型的对象，为整数值 19(对应于 MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar 和 2130708361(与 MediaCodecInfo.CodecCapabilities).
For <2>, I retrieved the MediaCodecInfo.CodecCapabilities-object for type video/avc, being the integer values 19 (corresponding with MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar and 2130708361 (which doesn't correspond with any value of MediaCodecInfo.CodecCapabilities).
除上述之外的任何其他值都会导致崩溃.
Any other value than the above results in a crash.
组合这些设置会产生不同的结果，我将在下面展示.这是 Android 上的屏幕截图(即真实"颜色):以下是 Gstreamer 显示的结果:
Combining these settings gives different results, which I'll show below. Here's the screenshot on Android (i.e. the "real" colors): Here are the results as shown by Gstreamer:
<1> = NV21, <2> = COLOR_FormatYUV420Planar
<1> = NV21, <2> = COLOR_FormatYUV420Planar
<1> = NV21，<2> = 2130708361
<1> = NV21, <2> = 2130708361
<1> = YV2, <2> = COLOR_FormatYUV420Planar
<1> = YV2, <2> = COLOR_FormatYUV420Planar
<1> = YV2，<2> = 2130708361
<1> = YV2, <2> = 2130708361
可以看出，这些都不令人满意.YV2 颜色空间看起来最有前途，但看起来红色 (Cr) 和蓝色 (Cb) 是颠倒的.我猜 NV21 看起来是交错的(不过，我不是这方面的专家).
As can be seen, none of these are satisfying. The YV2-colorspace looks the most promising, but it looks like red (Cr) and blue (Cb) are inverted. The NV21 looks interlaced I guess (however, I'm no expert in this field).
由于目的是与 Skype 通信，我认为我不应该更改解码器(即 Gstreamer 命令)，对吗?这是否要在 Android 中解决，如果是这样:如何解决?或者这可以通过添加某些 RTP 负载信息来解决吗?还有什么建议吗?
Since the purpose is to communicate with Skype, I assume I shouldn't change the decoder (i.e. the Gstreamer-command), right? Is this to be solved in Android and if so: how? Or can this be solved by adding certain RTP payload information? Any other suggestion?
推荐答案
我通过在 Android 级别自己交换字节平面来解决它，使用一个简单的函数:
I solved it by swapping the byteplanes myself on Android level, using a simple function:
public byte[] swapYV12toI420(byte[] yv12bytes, int width, int height) {
byte[] i420bytes = new byte[yv12bytes.length];
for (int i = 0; i < width*height; i++)
i420bytes[i] = yv12bytes[i];
for (int i = width*height; i < width*height + (width/2*height/2); i++)
i420bytes[i] = yv12bytes[i + (width/2*height/2)];
for (int i = width*height + (width/2*height/2); i < width*height + 2*(width/2*height/2); i++)
i420bytes[i] = yv12bytes[i - (width/2*height/2)];
return i420bytes;
}
这篇关于MediaCodec 和相机:色彩空间不匹配的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
