如何知道H.264流中代表图片的NAL单元数 [英] How to know the number of NAL unit in H.264 stream which represent a picture
问题描述
我正在使用通过avcodec解码的RTSP上具有H.264流的摄像机.对于大多数相机，接收到的每个数据包(NAL单元)代表一个完整的帧(I帧或帧)，当我对其进行解码时，每次都会获得一个帧.但是对于另一台摄像机，一帧被分成许多大小不变的NAL单元，当我解码每个数据包时，我没有每个数据包的帧.
I'm using cameras with an H.264 stream over RTSP decoded with avcodec. For most of the camera, each packet (NAL unit) received represent a complete frame (I Frame or frame) and when i decode it i obtain a frame each time. But for another camera, a frame is split in many NAL unit of constant size, when i decode each packet, i don't got a frame for each packet.
我看到NAL单元中有一个开始和结束标志.除了PPS或SPS，永远不会设置结束标志.不过，我可以检测到开始代码，并在新帧开始时告诉每个帧结束.
I saw that there is a start and end flags in NAL unit. The end flags is never set, except for PPS or SPS. Nevertheless, I can detect start code, and tell every frame end when a new frame begin.
我想在单个帧内缓冲每个NAL单元，然后再将其发送到解码器(这是用于记录功能并最大程度地减少帧索引).
I would like to buffer every NAL unit, within a single frame before to send it to the decoder (this for recording feature and to minimize frame indexing).
这里有个例子(NAL_1字节内的start_flags为128)
Here an example (start_flags is 128 within NAL[1] byte)
NALU: 10 bytes: SPS, NAL[1]={0,64,0,2} // Start Frame 1
NALU: 5 bytes: PPS, NAL[1]={128,64,0,14}
NALU: 551 bytes: I-Frame, NAL[1]={128,0,0,8}
NALU: 531 bytes: I-Frame, NAL[1]={0,0,0,9}
NALU: 532 bytes: I-Frame, NAL[1]={0,0,0,4}
NALU: 517 bytes: I-Frame, NAL[1]={0,0,0,7}
NALU: 533 bytes: I-Frame, NAL[1]={0,0,0,3}
NALU: 621 bytes: I-Frame, NAL[1]={0,0,0,3}
NALU: 586 bytes: I-Frame, NAL[1]={0,0,0,1}
NALU: 520 bytes: I-Frame, NAL[1]={0,0,0,1}
NALU: 507 bytes: I-Frame, NAL[1]={0,0,0,1}
NALU: 508 bytes: I-Frame, NAL[1]={0,0,0,1}
NALU: 531 bytes: I-Frame, NAL[1]={0,0,0,0}
NALU: 558 bytes: I-Frame, NAL[1]={0,0,0,0}
NALU: 49 bytes: I-Frame, NAL[1]={0,0,0,0} // Start Frame 2 + END Frame 1
NALU: 253 bytes: Frame, NAL[1]={128,0,0,26} // Start Frame 3 + END Frame 2
// Frame 2 start so we can record/decode Frame 1
NALU: 510 bytes: Frame, NAL[1]={128,0,0,26}
// Frame 3 start so we can record/decode Frame 2
NALU: 506 bytes: Frame, NAL[1]={0,0,0,1}
NALU: 267 bytes: Frame, NAL[1]={0,0,0,0} // Start Frame 4 + END Frame 3
NALU: 535 bytes: Frame, NAL[1]={128,0,0,26}
// Frame 4 start so we can record/decode Frame 3
NALU: 527 bytes: Frame, NAL[1]={0,0,0,4}
NALU: 509 bytes: Frame, NAL[1]={0,0,0,3}
NALU: 508 bytes: Frame, NAL[1]={0,0,0,1}
NALU: 519 bytes: Frame, NAL[1]={0,0,0,0}
NALU: 327 bytes: Frame, NAL[1]={0,0,0,0} // END Frame 4
...
但是，似乎我在某些视频流方面遇到了一些麻烦.对于其中每个NAL单元代表一个帧的流，如果我仅在下一次启动时才解码帧，则RTSP流似乎会丢失一些I帧.我认为这是由于可能需要解码时间而导致的同步问题，因为在直接接收时我解码帧不会发生该问题.
However, it seems i got some troubles with some streams. For stream where each NAL unit represent a frame, it seems if i decode frame only when the next start, the RTSP stream seems to drop some I-Frame. I think it's a synchronization problem due to the maybe due decoding time because the problem doesn't occurs is i decode frame when directly received.
这里是我直接解码时的细节(一切正常):
Here the detail when I decode directly (every thing works correctly):
NALU: 24 bytes: SPS, NAL[1]={0,64,0,13} // Start Frame 1
NALU: 4 bytes: PPS, NAL[1]={128,64,32,14}
NALU: 176124 bytes: Frame, NAL[1]={128,0,0,8}
// Decode Frame 1 OK
NALU: 24 bytes: SPS, NAL[1]={0,64,0,13} // Start Frame 2
NALU: 4 bytes: PPS, NAL[1]={128,64,32,14}
NALU: 175605 bytes: I-Frame, NAL[1]={128,0,0,8}
// Decode Frame 2 OK
NALU: 38777 bytes: Frame, NAL[1]={128,0,0,26} // Start Frame 3
// Decode Frame 3 OK
NALU: 32188 bytes: Frame, NAL[1]={128,0,0,26} // Start Frame 4
// Decode Frame 4 OK
NALU: 24 bytes: SPS, NAL[1]={0,64,0,13} // Start Frame 5
NALU: 4 bytes: PPS, NAL[1]={128,64,32,14}
NALU: 175975 bytes: I-Frame, NAL[1]={128,0,0,8}
// Decode Frame 5 OK
NALU: 41681 bytes: Frame, NAL[1]={128,0,0,26} // Start Frame 6
// Decode Frame 6 OK
在每帧开始后解码时，这里是细节(某些帧未解码):
Here the detail when I decode after each frame start (some frame are not decoded):
NALU: 24 bytes: NAL[0]={0,3,7}, NAL[1]={0,64,0,13} // Start frame 1
NALU: 4 bytes: NAL[0]={0,3,8}, NAL[1]={128,64,32,14}
NALU: 177827 bytes: NAL[0]={0,3,1}, NAL[1]={128,0,0,8}
NALU: 24 bytes: NAL[0]={0,3,7}, NAL[1]={0,64,0,13} // Start frame 2 + End frame 1
// Decode Frame 1 OK
NALU: 4 bytes: NAL[0]={0,3,8}, NAL[1]={128,64,32,14}
NALU: 43304 bytes: NAL[0]={0,3,1}, NAL[1]={128,0,0,26}
NALU: 39115 bytes: NAL[0]={0,3,1}, NAL[1]={128,0,0,26} // Start frame 3 + End frame 2
// Decode Frame 2 OK
NALU: 24 bytes: NAL[0]={0,3,7}, NAL[1]={0,64,0,13} // Start frame 4 + End frame 3
// Decode Frame 3 OK
NALU: 4 bytes: NAL[0]={0,3,8}, NAL[1]={128,64,32,14}
NALU: 49200 bytes: NAL[0]={0,3,1}, NAL[1]={128,0,0,26}
NALU: 41002 bytes: NAL[0]={0,3,1}, NAL[1]={128,0,0,26} // Start frame 5 + End frame 4
// Decode Frame 4 failed
NALU: 39581 bytes: NAL[0]={0,3,1}, NAL[1]={128,0,0,26}
// Decode Frame 5 failed
这就像RTSP流(I帧)是否丢弃了某些帧
It's like if some frame are dropped by the RTSP stream (I-Frame)
所以我的问题是:
- 您认为RTPS丢了一些帧吗?
- H.264解码器是否期望在某个延迟内到达的帧能够正确解码，并遵守某些时间码或类似的内容?
- 如何检测NAL单元是图片的最后一个，而不是等待下一个的开始.
谢谢您的帮助
推荐答案
首先，没有结束标志".仅附件B的起始代码和其他格式的NALU大小(我相信RTP使用附件B).在H.264中，您所说的帧称为访问单元.对于每个访问单元，非视频编码层(非VCL)NALUS可选地位于VCL nalus之前.因此，要确定您是否拥有所有VCL Nalu，您必须解析每个NALU，以确定哪些宏块被编码到切片中.通过使用从SPS解析的数据，您可以确定每帧可能有多少个宏块.然后，一旦收到所有宏块，就可以解码该帧.
First off, there is no "end flag". Only start codes for annex B and NALU size for other formats (I believe RTP uses annex B). What you are calling a frame, in H.264, is called an access unit. For each access unit non Video coding layer (non-VCL) NALUS optionally precede VCL nalus. So to determine if you have all the VCL Nalus, you must parse each NALU to determine what macroblocks are encoded into the slice. By using data you received parsed form the SPS, you can determine how may macroblocks there are per frame. Then once you have received all the macroblocks, you can decode the frame.
这篇关于如何知道H.264流中代表图片的NAL单元数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
