WebRTC:编码器实现中的RTPFragmentationHeader是什么? [英] WebRTC: What is RTPFragmentationHeader in encoder implementation?

问题描述

我修改了h264_encoder_impl以使用基于nvidia网格的硬件编码器.这是通过用Nvidia API调用替换OpenH264特定的调用来完成的.编码流可以成功写入文件，但是写入encoded_image_的_buffer和_size是不够的，并且还需要填充RTPFragmentationHeader.

I have modified h264_encoder_impl to use nvidia grid based hardware encoder. This is done by replacing OpenH264 specific calls with Nvidia API calls. Encoded stream can be written to file successfully but writing _buffer and _size of encoded_image_ are not enough and RTPFragmentationHeader also needs to be filled.

// RtpFragmentize(EncodedImage* encoded_image,
//                       std::unique_ptr<uint8_t[]>* encoded_image_buffer,
//                       const VideoFrameBuffer& frame_buffer,
//                       SFrameBSInfo* info,
//                      RTPFragmentationHeader* frag_header)

// encode
openh264_->Encode(input, &info /*out*/);

// fragmentize ?
RtpFragmentize(&encoded_image_ /*out*/, &encoded_image_buffer_, *frame_buffer, 
               &info, &frag_header /*out*/); 

// ...

// send 
encoded_image_callback_->OnEncodedImage(encoded_image_, &codec_specific, &frag_header);

当前基于Openh264的实现在RTPFragmentize()中填充frag_header，而VP8则以不同的方式填充.我可以看到带有NAL untis和图层的内容，这些内容也可以计算encoded_image->_length，但是我不知道该怎么做.

Current Openh264 based implementation fills frag_header in RTPFragmentize() and VP8 fills it differently. I can see something with NAL untis and layers which also calculates encoded_image->_length but I have no idea how.

我在任何地方都找不到任何文档. VP8和OpenH264实现就是我所拥有的.

I can not find any documentation on it anywhere. VP8 and OpenH264 implementations is all I have.

那么RTPFragmentationHeader是什么?它有什么作用?什么是encoded_image->_length?使用自定义H264编码器时如何正确填充?我可以找到起始码，但是接下来呢?如何填补其所有成员?

So what is RTPFragmentationHeader? what does it do? What is encoded_image->_length? How to fill it correctly when using custom H264 encoder? I can find startcode but what next? How to fill all its members?

推荐答案

在h264_encoder_impl中经过RTPFragmentize()之后，我已经弄清楚了.

After going through RTPFragmentize() in h264_encoder_impl I have figured it out.

在一个编码帧中，有多个NALU.有不同的NALU，包括AUD，SPS(67)，PPS(68)和IDR.每个NALU由4个字节的起始代码00 00 00 01分隔.

In an encoded frame there are multiple NALUs. There are different NALUs including AUD, SPS (67), PPS (68) and IDR. Each NALU is separated by the 4 byte start code which is 00 00 00 01.

对于OpenH264，第一帧的标题看起来像这样

For OpenH264, header looked like this for the first frame

[00 00 00 01 67 42 c0 20 8c 8d 40 20 03 09 00 f0  
 88 46 a0 00 00 00 01 68 ce 3c 80]00 00 00 01 .. 

您可以看到以粗体显示的起始代码.只有方括号之间的字节属于标题，最后的起始代码用于帧数据.

You can see start code in bold. Only bytes between square brackets belong to header, last start code is for frame data.

RTPFragmentationHeader:

frag_header->fragmentationVectorSize = 3     // 2 fragments for header
                                             // 3rd fragment for frame buffer

frag_header->fragmentationOffset[0]  = 4     
frag_header->fragmentationLength[0]  = 15

frag_header->fragmentationOffset[1]  = 23    // 4 + 15 + sizeof(startcode)
frag_header->fragmentationLength[1]  = 4    

frag_header->fragmentationOffset[2]  = 31   
frag_header->fragmentationLength[2]  = 43218 // last fragment is frame buffer

下一帧总是只有一个片段，看起来像下面的

Next frames always had only one fragment which looked like following

00 00 00 01 67 b8 .. .. ..

encoded_image->_length是实际编码的帧缓冲区的大小，并且
encoded_image->_size是已编码帧缓冲区的最大大小.

encoded_image->_length is the size of actual encoded frame buffer and
encoded_image->_size is maximum size of an encoded frame buffer.

OpenH264 API给出了编码帧中的NALU数量，而API I仅使用提供的标头及其大小来计算片段，无论标头是否实际与帧一起添加.仅在帧字节中搜索标头的大小，即可正确计算碎片.

OpenH264 API gives number of NALUs in encoded frame which is used to calculate fragments while the API I was using only provided header and its size no matter if header is actually added with frame or not. Searching frame bytes only for size of header allowed correct calculation of fragmentation.

这样做最终发送了编码后的数据，并且已在客户端浏览器上正确解码.

Doing this finally sent the encoded data and it was decoded correctly on client browser.

更新:从本质上讲，我必须完全跳过RTPFragmentize()，因为它是专门为OpenH264制作的，并根据上述观察结果自己计算了frag_header.

Update: In essence, I had to skip RTPFragmentize() entirely, because it is made specifically for OpenH264, and calculate frag_header myself based on above observations.

这篇关于WebRTC:编码器实现中的RTPFragmentationHeader是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！