从CMBlockBuffer中提取h264 [英] Extracting h264 from CMBlockBuffer

问题描述

我正在使用Apple VideoTool Box（iOS）来压缩设备相机拍摄的原始帧。

我的回调正在使用包含CMBlockBuffer的CMSampleBufferRef对象进行调用。

CMBlockBuffer对象包含H264基本流，但没有找到任何方式获取基本流的指针。

当我打印到控制台CMSampleBufferRef对象我得到：

 （lldb）po blockBufferRefCMBlockBuffer 0x1701193e0 totalDataLength：4264 retainCount：1分配器：0x1957c2c80 subBlockCapacity：2 [0] 4264 bytes @ offset 128缓冲区参考：CMBlockBuffer 0x170119350 totalDataLength：4632 retainCount：1 allocator：0x1957c2c80 subBlockCapacity：2 [0] 4632 bytes @ offset 0内存块0x10295c000，4632字节（自定义V = 0 A = 0x0 F = 0x18498bb44 R = 0x0） 

似乎我设法获取指针的CMBlockBuffer对象正在附加另一个CMBlockBuferRef（4632字节）这是不可访问的。

任何人都可以发布如何访问H264 elemantry流？

谢谢！ / p>

解决方案

我一直在努力争取一段时间，终于想出了一切。

函数 CMBlockBufferGetDataPointer 可让您访问所需的所有数据，但还有一些不太明显的事情需要做将其转换为基本流。

AVCC与附件B格式

CMBlockBuffer中的数据存储在AVCC格式，而基本流通常遵循附件B规范（ here 是一个很好的概述两种格式）。在AVCC格式中，4个第一个字节包含NAL单元的长度（H264数据包的另一个字）。您需要使用4字节起始码替换此标头：0x00 0x00 0x00 0x01，它用作附件B基本流中NAL单元之间的分隔符（3字节版本0x00 0x00 0x01也可以正常工作）。

单个CMBlockBuffer中的多个NAL单元

下一个不是很明显的事情是单个CMBlockBuffer有时会包含多个NAL单元。苹果似乎在每个I帧NAL单元（也称为IDR）中添加了一个包含元数据的附加NAL单元（SEI）。这可能是为什么您在单个CMBlockBuffer对象中看到多个缓冲区。但是， CMBlockBufferGetDataPointer 函数为您提供了访问所有数据的单个指针。话虽如此，多个NAL单元的存在使AVCC头的转换变得复杂。现在，您实际上必须读取AVCC标题中包含的长度值以找到下一个NAL单元，并继续转换标题，直到达到缓冲区的结尾。

Big-Endian vs Little-Endian

下一个不是很明显的事情是，AVCC头是以Big-Endian格式存储的，而iOS是本机的Little-Endian。所以当你读取AVCC标题中包含的长度值时，首先将它传递给 CFSwapInt32BigToHost 函数。

SPS最后不太明显的是，CMBlockBuffer中的数据不包含参数NAL单元SPS和PPS，其中包含解码器的配置参数如配置文件，级别，分辨率，帧速率。这些作为元数据存储在样本缓冲区的格式描述中，可以通过函数 CMVideoFormatDescriptionGetH264ParameterSetAtIndex 访问。请注意，您必须在发送之前将起始码添加到这些NAL单元。 SPS和PPS NAL设备不必每个新的帧发送。解码器只需要读取一次，但通常会定期重新发送它们，例如在每个新的I帧NAL单元之前重新发送。

代码示例

下面是一个代码示例，考虑所有这些事情。

  static void videoFrameFinishedEncoding（void * outputCallbackRefCon，
 void * sourceFrameRefCon，
 OSStatus status，
 VTEncodeInfoFlags infoFlags，
 CMSampleBufferRef sampleBuffer）{
 //检查是否有任何错误编码
 if（status！= noErr）{
 NSLog（@Error encoding video，err =％lld，（int64_t）status）; 
 return; 
} 
 
 //在这个例子中，我们将使用一个NSMutableData对象来存储
 //基本流。 
 NSMutableData * elementaryStream = [NSMutableData data]; 
 
 
 //找出样本缓冲区是否包含I帧。 
 //如果是这样，我们将把SPS和PPS NAL单元写入基本流。 
 BOOL isIFrame = NO; 
 CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray（sampleBuffer，0）; 
 if（CFArrayGetCount（attachmentsArray））{
 CFBooleanRef notSync; 
 CFDictionaryRef dict = CFArrayGetValueAtIndex（attachmentsArray，0）; 
 BOOL keyExists = CFDictionaryGetValueIfPresent（dict，
 kCMSampleAttachmentKey_NotSync，
（const void **）& notSync）; 
 // I帧是同步帧
 isIFrame =！keyExists || ！CFBooleanGetValue（notSync）; 
} 
 
 //这是我们将写入
的开始代码//每个NAL单元之前的基本流
 static const size_t startCodeLength = 4; 
 static const uint8_t startCode [] = {0x00，0x00，0x00，0x01}; 
 
 //在每个I-Frame 
之前，将SPS和PPS NAL单元写入基本流if（isIFrame）{
 CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription（sampleBuffer）; 
 
 //找出有多少个参数集
 size_t numberOfParameterSets; 
 CMVideoFormatDescriptionGetH264ParameterSetAtIndex（description，
 0，NULL，NULL，
& numberOfParameterSets，
 NULL）; 
 
 //将每个参数集写入基本流
（int i = 0; i< numberOfParameterSets; i ++）{
 const uint8_t * parameterSetPointer; 
 size_t parameterSetLength; 
 CMVideoFormatDescriptionGetH264ParameterSetAtIndex（description，
 i，
& parameterSetPointer，
& parameterSetLength，
 NULL，NULL）; 
 
 //将参数集写入基本流
 [elementaryStream appendBytes：startCode length：startCodeLength]; 
 [elementaryStream appendBytes：parameterSetPointer length：parameterSetLength]; 
} 
} 
 
 //获取一个指向样本缓冲区中原始AVCC NAL单元数据的指针
 size_t blockBufferLength; 
 uint8_t * bufferDataPointer = NULL; 
 CMBlockBufferGetDataPointer（CMSampleBufferGetDataBuffer（sampleBuffer），
 0，
 NULL，
& blockBufferLength，
（char **）& bufferDataPointer）; 
 
 //循环遍历块缓冲区
 //中的所有NAL单元，并将它们写入基本流，
 //开始代码而不是AVCC长度头
 size_t bufferOffset = 0; 
 static const int AVCCHeaderLength = 4; 
 while（bufferOffset< blockBufferLength  -  AVCCHeaderLength）{
 //读取NAL单元长度
 uint32_t NALUnitLength = 0; 
 memcpy（& NALUnitLength，bufferDataPointer + bufferOffset，AVCCHeaderLength）; 
 //将长度值从Big-endian转换为Little-endian 
 NALUnitLength = CFSwapInt32BigToHost（NALUnitLength）; 
 //将开始代码写入基本流
 [elementaryStream appendBytes：startCode length：startCodeLength]; 
 //将没有AVCC长度头的NAL单元写入基本流
 [elementaryStream appendBytes：bufferDataPointer + bufferOffset + AVCCHeaderLength 
 length：NALUnitLength]; 
 //移动到块缓冲区中的下一个NAL单元
 bufferOffset + = AVCCHeaderLength + NALUnitLength; 
} 
} 
  

I am using Apple VideoTool Box (iOS) to compress raw frames captured by the device camera.

My callback is being called with a CMSampleBufferRef object that contains CMBlockBuffer.

The CMBlockBuffer object contain the H264 elementary stream but I didn't find any way to get a pointer to the elementary stream.

When I printed into the console the CMSampleBufferRef object I got:

(lldb) po blockBufferRef
CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
 [0] 4264 bytes @ offset 128 Buffer Reference:
    CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
     [0] 4632 bytes @ offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0)

It seems that the CMBlockBuffer object that I managed to get pointer to is contaning another CMBlockBuferRef (4632 bytes) which is not accessible.

Can anyone post how to access the H264 elemantry stream?

Thank you!

解决方案

I've been struggling with this myself for quite some time now, and have finally figured everything out.

The function CMBlockBufferGetDataPointer gives you access to all the data you need, but there are a few not very obvious things you need to do to convert it to an elementary stream.

AVCC vs Annex B format

The data in the CMBlockBuffer is stored in AVCC format, while elementary streams are typically following the Annex B specification (here is an excellent overview of the two formats). In the AVCC format, the 4 first bytes contains the length of the NAL unit (another word for H264 packet). You need to replace this header with the 4 byte start code: 0x00 0x00 0x00 0x01, which functions as a separator between NAL units in an Annex B elementary stream (the 3 byte version 0x00 0x00 0x01 works fine too).

Multiple NAL units in a single CMBlockBuffer

The next not very obvious thing is that a single CMBlockBuffer will sometimes contain multiple NAL units. Apple seems to add an additional NAL unit (SEI) containing metadata to every I-Frame NAL unit (also called IDR). This is probably why you are seeing multiple buffers in a single CMBlockBuffer object. However, the CMBlockBufferGetDataPointer function gives you a single pointer with access to all the data. That being said, the presence of multiple NAL units complicates the conversion of the AVCC headers. Now you actually have to read the length value contained in the AVCC header to find the next NAL unit, and continue converting headers until you have reached the end of the buffer.

Big-Endian vs Little-Endian

The next not very obvious thing is that the AVCC header is stored in Big-Endian format, and iOS is Little-Endian natively. So when you are reading the length value contained in an AVCC header pass it to the CFSwapInt32BigToHost function first.

SPS and PPS NAL units

The final not very obvious thing is that the data inside the CMBlockBuffer does not contain the parameter NAL units SPS and PPS, which contains configuration parameters for the decoder such as profile, level, resolution, frame rate. These are stored as metadata in the sample buffer's format description and can be accessed via the function CMVideoFormatDescriptionGetH264ParameterSetAtIndex. Note that you have to add the start codes to these NAL units before sending. The SPS and PPS NAL units does not have to be sent with every new frame. A decoder only needs to read them once, but it is common to resend them periodically, for example before every new I-frame NAL unit.

Code Example

Below is a code example taking all of these things into account.

static void videoFrameFinishedEncoding(void *outputCallbackRefCon,
                                       void *sourceFrameRefCon,
                                       OSStatus status,
                                       VTEncodeInfoFlags infoFlags,
                                       CMSampleBufferRef sampleBuffer) {
    // Check if there were any errors encoding
    if (status != noErr) {
        NSLog(@"Error encoding video, err=%lld", (int64_t)status);
        return;
    }

    // In this example we will use a NSMutableData object to store the
    // elementary stream.
    NSMutableData *elementaryStream = [NSMutableData data];


    // Find out if the sample buffer contains an I-Frame.
    // If so we will write the SPS and PPS NAL units to the elementary stream.
    BOOL isIFrame = NO;
    CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
    if (CFArrayGetCount(attachmentsArray)) {
        CFBooleanRef notSync;
        CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0);
        BOOL keyExists = CFDictionaryGetValueIfPresent(dict,
                                                       kCMSampleAttachmentKey_NotSync,
                                                       (const void **)&notSync);
        // An I-Frame is a sync frame
        isIFrame = !keyExists || !CFBooleanGetValue(notSync);
    }

    // This is the start code that we will write to
    // the elementary stream before every NAL unit
    static const size_t startCodeLength = 4;
    static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01};

    // Write the SPS and PPS NAL units to the elementary stream before every I-Frame
    if (isIFrame) {
        CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer);

        // Find out how many parameter sets there are
        size_t numberOfParameterSets;
        CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
                                                           0, NULL, NULL,
                                                           &numberOfParameterSets,
                                                           NULL);

        // Write each parameter set to the elementary stream
        for (int i = 0; i < numberOfParameterSets; i++) {
            const uint8_t *parameterSetPointer;
            size_t parameterSetLength;
            CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
                                                               i,
                                                               &parameterSetPointer,
                                                               &parameterSetLength,
                                                               NULL, NULL);

            // Write the parameter set to the elementary stream
            [elementaryStream appendBytes:startCode length:startCodeLength];
            [elementaryStream appendBytes:parameterSetPointer length:parameterSetLength];
        }
    }

    // Get a pointer to the raw AVCC NAL unit data in the sample buffer
    size_t blockBufferLength;
    uint8_t *bufferDataPointer = NULL;
    CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer),
                                0,
                                NULL,
                                &blockBufferLength,
                                (char **)&bufferDataPointer);

    // Loop through all the NAL units in the block buffer
    // and write them to the elementary stream with
    // start codes instead of AVCC length headers
    size_t bufferOffset = 0;
    static const int AVCCHeaderLength = 4;
    while (bufferOffset < blockBufferLength - AVCCHeaderLength) {
        // Read the NAL unit length
        uint32_t NALUnitLength = 0;
        memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength);
        // Convert the length value from Big-endian to Little-endian
        NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
        // Write start code to the elementary stream
        [elementaryStream appendBytes:startCode length:startCodeLength];
        // Write the NAL unit without the AVCC length header to the elementary stream
        [elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength
                               length:NALUnitLength];
        // Move to the next NAL unit in the block buffer
        bufferOffset += AVCCHeaderLength + NALUnitLength;
    }
}   

这篇关于从CMBlockBuffer中提取h264的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！