视频渲染损坏 MediaCodec H.264 流 [英] Video rendering is broken MediaCodec H.264 stream
问题描述
我正在使用 MediaCodec Java API 实现解码器，用于解码实时 H.264 远程流.我正在使用回调 (void OnRecvEncodedData(byte[] encodingData)) 从本机层接收 H.264 编码数据，在 TextureView<的 Surface 上解码和呈现/代码>.我的实现已经完成(使用回调、解码和渲染等检索编码流).这是我的解码器类:
public class MediaCodecDecoder extends Thread 实现 MyFrameAvailableListener {私有静态最终布尔值 VERBOSE = true;私有静态最终字符串LOG_TAG = MediaCodecDecoder.class.getSimpleName();private static final String VIDEO_FORMAT = "video/avc";//h.264私有静态最终长 mTimeoutUs = 10000l;私有 MediaCodec mMediaCodec；表面 mSurface;易失性布尔 m_bConfigured;易失性布尔 m_bRunning;漫长的开始；公共 MediaCodecDecoder() {JniWrapper.SetFrameAvailableListener(this);}//这是我从本机层接收编码流的回调@覆盖公共无效OnRecvEncodedData(字节[]编码数据){if(!m_bConfigured && bKeyFrame(encodedData)) {配置(mSurface, 240, 320, 编码数据);}如果(m_bConfigured){解码数据(编码数据)；}}公共无效SetSurface(表面表面){如果(mSurface == null){mSurface = 表面；}}公共无效开始(){如果(m_b运行)返回;m_bRunning = 真；开始();}公共无效停止(){如果(！m_b运行)返回;m_bRunning = false;mMediaCodec.stop();mMediaCodec.release();}私有无效配置(表面，整数宽度，整数高度，字节[] csd0){如果(m_bConfigured){Log.e(LOG_TAG, "解码器已经配置");返回;}如果(mSurface == null){Log.d(LOG_TAG, "表面不可用/尚未设置.");返回;}MediaFormat 格式 = MediaFormat.createVideoFormat(VIDEO_FORMAT, width, height);format.setByteBuffer("csd-0", ByteBuffer.wrap(csd0));尝试 {mMediaCodec = MediaCodec.createDecoderByType(VIDEO_FORMAT);} catch (IOException e) {Log.d(LOG_TAG, "创建编解码器失败:" + e.getMessage());}startMs = System.currentTimeMillis();mMediaCodec.configure(格式，表面，空，0)；if (VERBOSE) Log.d(LOG_TAG, "解码器配置.");mMediaCodec.start();Log.d(LOG_TAG, "解码器初始化.");m_bConfigured = true;}@SuppressWarnings("弃用")私有无效解码数据(字节 [] 数据){如果(！m_bConfigured){Log.e(LOG_TAG, "解码器尚未配置.");返回;}int inIndex = mMediaCodec.dequeueInputBuffer(mTimeoutUs);如果 (inIndex >= 0) {ByteBuffer 缓冲区；如果 (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {buffer = mMediaCodec.getInputBuffers()[inIndex];缓冲区清除()；} 别的 {缓冲区 = mMediaCodec.getInputBuffer(inIndex);}如果(缓冲区！= null){缓冲区.放置(数据)；longpresentationTimeUs = System.currentTimeMillis() - startMs;mMediaCodec.queueInputBuffer(inIndex, 0, data.length,presentationTimeUs, 0);}}}私有静态布尔 bKeyFrame(byte[] frameData) {返回 ( ( (frameData[4] & 0xFF) & 0x0F) == 0x07);}@覆盖公共无效运行(){尝试 {MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();而(m_bRunning){如果(m_bConfigured){int outIndex = mMediaCodec.dequeueOutputBuffer(info, mTimeoutUs);if(outIndex >= 0) {mMediaCodec.releaseOutputBuffer(outIndex, true);}} 别的 {尝试 {线程睡眠(10)；} catch (InterruptedException 忽略) {}}}} 最后 {停止();}}}现在的问题是 - 流正在被解码并呈现在表面上，但视频不清楚.似乎框架损坏了，场景失真/脏了.运动破碎，到处都是方形碎片(我真的很抱歉，因为我现在没有截图).
关于我的流 - 它是 H.264 编码的，仅包含 I 帧和 P 帧(没有 B 帧).每个 I 帧都有 SPS + PPS + payload 结构.编码时使用的颜色格式(在native层使用FFMPEG)是YUV420 planner.native层发送的数据长度是可以的(宽*高*(3/2)).
在 configure() 期间，我只是用 SPS 框架设置了 csd-0 值.配置使用的帧是I帧(SPS+PPS+payload)——前缀是SPS帧，所以我认为配置成功.请注意，我没有使用 PPS 框架设置 csd-1 值(有问题吗?).
对于 p 帧和 I 帧(对于 I 帧，起始代码都出现在 SPS 和 PPS 帧的前面)，每个帧都有前面的起始代码 (0x00 0x00 0x00 0x01).
此外，我将每个帧的演示时间戳设置为 System.currrentTimeMillis() - startTime，这是每个新帧递增的顺序.我认为这不会造成任何问题(如果我错了，请纠正我).
我的设备是 Google 的 Nexus 5，Android 版本为 4.4.4，芯片组是 Qualcomm MSM8974 Snapdragon 800.我使用 Surface 进行解码，所以我认为不应该有任何设备特定的颜色格式不匹配问题.
如果需要，我也可以提供我的 TextureView 代码.
我的解码/渲染不正确的原因可能是什么?提前致谢！
编辑 1
我尝试在配置期间手动传递特定于编解码器的数据(SPS 和 PPS 字节).但这并没有做出任何改变:(
byte[] sps = {0x00, 0x00, 0x00, 0x01, 0x67, 0x4d, 0x40, 0x0c, (byte) 0xda, 0x0f, 0x0a, 0x68, 0x40, 0x,00300, 0x00, 0x00, 0x07, (字节) 0xa3, (字节) 0xc5, 0x0a, (字节) 0xa8};format.setByteBuffer("csd-0", ByteBuffer.wrap(sps));byte[] pps = {0x00, 0x00, 0x00, 0x01, 0x68, (byte) 0xef, 0x04, (byte) 0xf2, 0x00, 0x00};format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));我也尝试过修剪起始代码 (0x00, 0x00, 0x00, 0x01) 但没有进展！
编辑 2
我尝试使用硬件加速 {{TextureView}}，正如
编辑 4
为了进一步说明，这是我的 H.264 编码的 I 帧十六进制流格式:
<块引用>00 00 00 01 67 4d 40 0c da 0f 0a 68 40 00 00 03 00 40 00 00 07 a3 c50a a8 00 00 00 01 68 ef 04 f2 00 00 01 06 05 ff ff 69 dc 45 e9 bd e6d9 48 b7 96 2c d8 20 d9 23 ee ef 78 32 36 34 20 2d 20 63 6f 72 65 2031 34 36 20 2d 20 48 2e 32 36 34 2f 4d 50 45 47 2d 34 20 41 56 43 2063 6f 64 65 63 20 2d 20 43 6f 70 79 6c 65 66 74 20 32 30 30 33 2d 3230 31 35 20 2d 20 68 74 74 70 3a 2f 2f 77 77 77 2e 76 69 64 65 6f 6c61 6e 2e 6f 72 67 2f 78 32 36 34 2e 68 74 6d 6c 20 2d 20 6f 70 74 696f 6e 73 3a 20 63 61 62 61 63 3d 31 20 72 65 66 3d 31 20 64 65 62 6c6f 63 6b 3d 31 3a 30 3a 30 20 61 6e 61 6c 79 73 65 3d 30 78 31 3a 3078 31 20 6d 65 3d 68 65 78 20 73 75 62 6d 65 3d 30 20 70 73 79 3d 3120 70 73 79 5f 72 64 3d 31 2e 30 30 3a 30 2e 30 30 20 6d 69 78 65 645f 72 65 66 3d 30 20 6d 65 5f 72 61 6e 67 65 3d 31 36 20 63 68 72 6f6d 61 5f 6d 65 3d 31 20 74 72 65 6c 6c 69 73 3d 30 20 38 78 38 64 6374
这是一个P帧:
<块引用>00 00 00 01 41 9a 26 22 df 76 4b b2 ef cf 57 ac 5b b6 3b 68 b9 87 b271 a5 9b 61 3c 93 47 bc 79 c5 ab 0f 87 34 f6 40 6a cd 80 03 b1 a2 c24e 08 13 cd 4e 3c 62 3e 44 0a e8 97 80 ec 81 3f 31 7c f1 29 f1 43 a0c0 a9 0a 74 62 c7 62 74 da c3 94 f5 19 23 ff 4b 9c c1 69 55 54 2f 62f0 5e 64 7f 18 3f 58 73 af 93 6e 92 06 fd 9f a1 1a 80 cf 86 71 24 7df7 56 2c c1 57 cf ba 05 17 77 18 f1 8b 3c 33 40 18 30 1f b0 19 23 44ec 91 c4 bd 80 65 4a 46 b3 1e 53 5d 6d a3 f0 b5 50 3a 93 ba 81 71 f309 98 41 43 ba 5f a1 0d 41 a3 7b c3 fd eb 15 89 75 66 a9 ee 3a 9c 1bc1 aa f8 58 10 88 0c 79 77 ff 7d 15 28 eb 12 a7 1b 76 36 aa 84 e1 3e63 cf a9 a3 cf 4a 2d c2 33 18 91 30 f7 3c 9c 56 f5 4c 12 6c 4b 12 1fc5 ec 5a 98 8c 12 75 eb fd 98 a4 fb 7f 80 5d 28 f9 ef 43 a4 0a ca 2575 19 6b f7 14 7b 76 af e9 8f 7d 79 fa 9d 9a 63 de 1f be fa 6c 65 ba5f 9d b0 b0 f4 71 cb e2 ea d6 dc c6 55 98 1b cd 55 d9 eb 9c 75 fc 9dec
我非常确定我的流的正确性，因为我使用 ffmpeg 解码和 GLSurfaceview 和 OpenGLES 2.0 成功渲染.
我从native层和Java层都拿了H.264转储，发现native层的转储播放完美，但Java层的转储播放坏了解码后的流.问题是 - 在将编码流从本机层传递到 Java 的过程中，编码流没有正确传递(损坏)，这是因为我的错误实现(抱歉，因为给您带来的不便，谁关注了这个线程).
此外，我仅将 I 帧的有效载荷传递给导致渲染中断的解码器.现在我正在通过完整的 NAL 单元(SPS + PPS + 有效载荷)，现在一切正常:)
I am implementing a decoder using MediaCodec Java API for decoding live H.264 remote stream. I am receiving H.264 encoded data from native layer using a callback (void OnRecvEncodedData(byte[] encodedData)), decode and render on Surface of TextureView. My implementation is completed (retrieving encoded streams using callback, decode and rendering etc). Here is my decoder class:
public class MediaCodecDecoder extends Thread implements MyFrameAvailableListener {
private static final boolean VERBOSE = true;
private static final String LOG_TAG = MediaCodecDecoder.class.getSimpleName();
private static final String VIDEO_FORMAT = "video/avc"; // h.264
private static final long mTimeoutUs = 10000l;
private MediaCodec mMediaCodec;
Surface mSurface;
volatile boolean m_bConfigured;
volatile boolean m_bRunning;
long startMs;
public MediaCodecDecoder() {
JniWrapper.SetFrameAvailableListener(this);
}
// this is my callback where I am receiving encoded streams from native layer
@Override
public void OnRecvEncodedData(byte[] encodedData) {
if(!m_bConfigured && bKeyFrame(encodedData)) {
Configure(mSurface, 240, 320, encodedData);
}
if(m_bConfigured) {
decodeData(encodedData);
}
}
public void SetSurface(Surface surface) {
if (mSurface == null) {
mSurface = surface;
}
}
public void Start() {
if(m_bRunning)
return;
m_bRunning = true;
start();
}
public void Stop() {
if(!m_bRunning)
return;
m_bRunning = false;
mMediaCodec.stop();
mMediaCodec.release();
}
private void Configure(Surface surface, int width, int height, byte[] csd0) {
if (m_bConfigured) {
Log.e(LOG_TAG, "Decoder is already configured");
return;
}
if (mSurface == null) {
Log.d(LOG_TAG, "Surface is not available/set yet.");
return;
}
MediaFormat format = MediaFormat.createVideoFormat(VIDEO_FORMAT, width, height);
format.setByteBuffer("csd-0", ByteBuffer.wrap(csd0));
try {
mMediaCodec = MediaCodec.createDecoderByType(VIDEO_FORMAT);
} catch (IOException e) {
Log.d(LOG_TAG, "Failed to create codec: " + e.getMessage());
}
startMs = System.currentTimeMillis();
mMediaCodec.configure(format, surface, null, 0);
if (VERBOSE) Log.d(LOG_TAG, "Decoder configured.");
mMediaCodec.start();
Log.d(LOG_TAG, "Decoder initialized.");
m_bConfigured = true;
}
@SuppressWarnings("deprecation")
private void decodeData(byte[] data) {
if (!m_bConfigured) {
Log.e(LOG_TAG, "Decoder is not configured yet.");
return;
}
int inIndex = mMediaCodec.dequeueInputBuffer(mTimeoutUs);
if (inIndex >= 0) {
ByteBuffer buffer;
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
buffer = mMediaCodec.getInputBuffers()[inIndex];
buffer.clear();
} else {
buffer = mMediaCodec.getInputBuffer(inIndex);
}
if (buffer != null) {
buffer.put(data);
long presentationTimeUs = System.currentTimeMillis() - startMs;
mMediaCodec.queueInputBuffer(inIndex, 0, data.length, presentationTimeUs, 0);
}
}
}
private static boolean bKeyFrame(byte[] frameData) {
return ( ( (frameData[4] & 0xFF) & 0x0F) == 0x07);
}
@Override
public void run() {
try {
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
while(m_bRunning) {
if(m_bConfigured) {
int outIndex = mMediaCodec.dequeueOutputBuffer(info, mTimeoutUs);
if(outIndex >= 0) {
mMediaCodec.releaseOutputBuffer(outIndex, true);
}
} else {
try {
Thread.sleep(10);
} catch (InterruptedException ignore) {
}
}
}
} finally {
Stop();
}
}
}
Now the problem is - the streams is being decoded and rendered on surface but the video is not clear. It seems like the frames are broken and scene is distorted/dirty. The movement is broken and square shaped fragments everywhere (I am really sorry as I don't have the screenshot right now).
About my streams - its H.264 encoded and consists of I frames and P frames only (there is no B frame). Every I frame has SPS + PPS + payload structure. The color format used during encoding (using FFMPEG in native layer) is YUV420 planner. The sent length of data from native layer is okay (width * height * (3 / 2)).
During configure() I just set the csd-0 value with SPS frame. The frame used for configuration was an I frame (SPS + PPS + payload) - the prefix was a SPS frame, so I think the configuration was successful. Note that, I didn't set the csd-1 value with PPS frame (is it a problem?).
Every frame has preceding start codes (0x00 0x00 0x00 0x01) for both p-frame and I-frame (for I-frame the start code is present both infront of SPS and PPS frame).
Moreover, I am setting the presentation timestamp as System.currrentTimeMillis() - startTime for every frame which is increasing order for every new frame. I think this shouldn't cause any problem (Correct me if I am wrong).
My device is Nexus 5 from Google with Android version 4.4.4 and chipset is Qualcomm MSM8974 Snapdragon 800. I am using Surface for decoding, so I think there should not be any device specific color format mismatch issues.
I can also provide my TextureView code if needed.
What might be the cause of my incorrect decoding/rendering? Thanks in advance!
EDIT 1
I tried by manually passing my codec-specific data(SPS and PPS bytes) during configuration. But this didn't make any change :(
byte[] sps = {0x00, 0x00, 0x00, 0x01, 0x67, 0x4d, 0x40, 0x0c, (byte) 0xda, 0x0f, 0x0a, 0x68, 0x40, 0x00, 0x00, 0x03, 0x00, 0x40, 0x00, 0x00, 0x07, (byte) 0xa3, (byte) 0xc5, 0x0a, (byte) 0xa8};
format.setByteBuffer("csd-0", ByteBuffer.wrap(sps));
byte[] pps = {0x00, 0x00, 0x00, 0x01, 0x68, (byte) 0xef, 0x04, (byte) 0xf2, 0x00, 0x00};
format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
I also tried by trimming the start codes (0x00, 0x00, 0x00, 0x01) but no progress!
EDIT 2
I tried with hardware accelerated {{TextureView}} as it is mentioned in official documentation (though I didn't find any H/W acceleration code in sample project of MediaCodec-textureView). But still no progress. Now I commented the H/W acceleration code snippet.
EDIT 3
The screenshots are avilable now:
EDIT 4
For further clarification, this is my H.264 encoded I-frame hex stream format:
00 00 00 01 67 4d 40 0c da 0f 0a 68 40 00 00 03 00 40 00 00 07 a3 c5 0a a8 00 00 00 01 68 ef 04 f2 00 00 01 06 05 ff ff 69 dc 45 e9 bd e6 d9 48 b7 96 2c d8 20 d9 23 ee ef 78 32 36 34 20 2d 20 63 6f 72 65 20 31 34 36 20 2d 20 48 2e 32 36 34 2f 4d 50 45 47 2d 34 20 41 56 43 20 63 6f 64 65 63 20 2d 20 43 6f 70 79 6c 65 66 74 20 32 30 30 33 2d 32 30 31 35 20 2d 20 68 74 74 70 3a 2f 2f 77 77 77 2e 76 69 64 65 6f 6c 61 6e 2e 6f 72 67 2f 78 32 36 34 2e 68 74 6d 6c 20 2d 20 6f 70 74 69 6f 6e 73 3a 20 63 61 62 61 63 3d 31 20 72 65 66 3d 31 20 64 65 62 6c 6f 63 6b 3d 31 3a 30 3a 30 20 61 6e 61 6c 79 73 65 3d 30 78 31 3a 30 78 31 20 6d 65 3d 68 65 78 20 73 75 62 6d 65 3d 30 20 70 73 79 3d 31 20 70 73 79 5f 72 64 3d 31 2e 30 30 3a 30 2e 30 30 20 6d 69 78 65 64 5f 72 65 66 3d 30 20 6d 65 5f 72 61 6e 67 65 3d 31 36 20 63 68 72 6f 6d 61 5f 6d 65 3d 31 20 74 72 65 6c 6c 69 73 3d 30 20 38 78 38 64 63 74
And this is a P-frame:
00 00 00 01 41 9a 26 22 df 76 4b b2 ef cf 57 ac 5b b6 3b 68 b9 87 b2 71 a5 9b 61 3c 93 47 bc 79 c5 ab 0f 87 34 f6 40 6a cd 80 03 b1 a2 c2 4e 08 13 cd 4e 3c 62 3e 44 0a e8 97 80 ec 81 3f 31 7c f1 29 f1 43 a0 c0 a9 0a 74 62 c7 62 74 da c3 94 f5 19 23 ff 4b 9c c1 69 55 54 2f 62 f0 5e 64 7f 18 3f 58 73 af 93 6e 92 06 fd 9f a1 1a 80 cf 86 71 24 7d f7 56 2c c1 57 cf ba 05 17 77 18 f1 8b 3c 33 40 18 30 1f b0 19 23 44 ec 91 c4 bd 80 65 4a 46 b3 1e 53 5d 6d a3 f0 b5 50 3a 93 ba 81 71 f3 09 98 41 43 ba 5f a1 0d 41 a3 7b c3 fd eb 15 89 75 66 a9 ee 3a 9c 1b c1 aa f8 58 10 88 0c 79 77 ff 7d 15 28 eb 12 a7 1b 76 36 aa 84 e1 3e 63 cf a9 a3 cf 4a 2d c2 33 18 91 30 f7 3c 9c 56 f5 4c 12 6c 4b 12 1f c5 ec 5a 98 8c 12 75 eb fd 98 a4 fb 7f 80 5d 28 f9 ef 43 a4 0a ca 25 75 19 6b f7 14 7b 76 af e9 8f 7d 79 fa 9d 9a 63 de 1f be fa 6c 65 ba 5f 9d b0 b0 f4 71 cb e2 ea d6 dc c6 55 98 1b cd 55 d9 eb 9c 75 fc 9d ec
I am pretty sure about my stream's correctness as I successfully rendered using ffmpeg decoding and GLSurfaceview with OpenGLES 2.0.
I took H.264 dump both from native layer and Java layer and found the dump of native layer was played perfectly but the Java layer's dump was played as broken as the decoded stream. The problem was - during passing encoded streams from native layer to Java, the encoded stream was not passed properly(corrupted) and it was because of my buggy implementation (sorry to who were following this thread for this inconvenience).
Moreover I was passing I-frame's payload only to decoder which resulted in broken rendering. Now I am passing complete NAL unit (SPS + PPS + payload) and everything is okay now :)
这篇关于视频渲染损坏 MediaCodec H.264 流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
