从 CMBlockBuffer 中提取 h264 [英] Extracting h264 from CMBlockBuffer
问题描述
我正在使用 Apple VideoTool Box (iOS) 来压缩设备摄像头捕获的原始帧.
I am using Apple VideoTool Box (iOS) to compress raw frames captured by the device camera.
正在使用包含 CMBlockBuffer 的 CMSampleBufferRef 对象调用我的回调.
My callback is being called with a CMSampleBufferRef object that contains CMBlockBuffer.
CMBlockBuffer 对象包含 H264 基本流，但我没有找到任何方法来获取指向基本流的指针.
The CMBlockBuffer object contain the H264 elementary stream but I didn't find any way to get a pointer to the elementary stream.
当我将 CMSampleBufferRef 对象打印到控制台时:
When I printed into the console the CMSampleBufferRef object I got:
(lldb) po blockBufferRef
CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4264 bytes @ offset 128 Buffer Reference:
CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4632 bytes @ offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0)我设法获得指针的 CMBlockBuffer 对象似乎包含另一个无法访问的 CMBlockBuferRef(4632 字节).
It seems that the CMBlockBuffer object that I managed to get pointer to is contaning another CMBlockBuferRef (4632 bytes) which is not accessible.
谁能发布如何访问 H264 元素流?
Can anyone post how to access the H264 elemantry stream?
谢谢！
推荐答案
我自己已经为此苦苦挣扎了很长一段时间，终于弄清楚了一切.
I've been struggling with this myself for quite some time now, and have finally figured everything out.
函数 CMBlockBufferGetDataPointer 使您可以访问所需的所有数据，但您需要做一些不太明显的事情才能将其转换为基本流.
The function CMBlockBufferGetDataPointer gives you access to all the data you need, but there are a few not very obvious things you need to do to convert it to an elementary stream.
CMBlockBuffer 中的数据以 AVCC 格式存储，而基本流通常遵循附件 B 规范(这里是对这两种格式的一个很好的概述).在 AVCC 格式中，前 4 个字节包含 NAL 单元的长度(H264 数据包的另一个词).您需要用 4 字节起始代码替换此标头:0x00 0x00 0x00 0x01，它用作 Annex B 基本流中 NAL 单元之间的分隔符(3 字节版本 0x00 0x00 0x01 也可以正常工作).
The data in the CMBlockBuffer is stored in AVCC format, while elementary streams are typically following the Annex B specification (here is an excellent overview of the two formats). In the AVCC format, the 4 first bytes contains the length of the NAL unit (another word for H264 packet). You need to replace this header with the 4 byte start code: 0x00 0x00 0x00 0x01, which functions as a separator between NAL units in an Annex B elementary stream (the 3 byte version 0x00 0x00 0x01 works fine too).
接下来不太明显的是，单个 CMBlockBuffer 有时会包含多个 NAL 单元.Apple 似乎向每个 I-Frame NAL 单元(也称为 IDR)添加了一个包含元数据的额外 NAL 单元(SEI).这可能就是您在单个 CMBlockBuffer 对象中看到多个缓冲区的原因.但是，CMBlockBufferGetDataPointer 函数为您提供了一个可以访问所有数据的指针.也就是说，多个 NAL 单元的存在使 AVCC 标头的转换复杂化.现在您实际上必须读取包含在 AVCC 标头中的长度值才能找到下一个 NAL 单元，并继续转换标头，直到到达缓冲区的末尾.
The next not very obvious thing is that a single CMBlockBuffer will sometimes contain multiple NAL units. Apple seems to add an additional NAL unit (SEI) containing metadata to every I-Frame NAL unit (also called IDR). This is probably why you are seeing multiple buffers in a single CMBlockBuffer object. However, the CMBlockBufferGetDataPointer function gives you a single pointer with access to all the data. That being said, the presence of multiple NAL units complicates the conversion of the AVCC headers. Now you actually have to read the length value contained in the AVCC header to find the next NAL unit, and continue converting headers until you have reached the end of the buffer.
接下来不太明显的是，AVCC头是以Big-Endian格式存储的，而iOS本身就是Little-Endian.因此，当您读取 AVCC 标头中包含的长度值时，首先将其传递给 CFSwapInt32BigToHost 函数.
The next not very obvious thing is that the AVCC header is stored in Big-Endian format, and iOS is Little-Endian natively. So when you are reading the length value contained in an AVCC header pass it to the CFSwapInt32BigToHost function first.
最后一个不太明显的就是CMBlockBuffer里面的数据不包含参数NAL单元SPS和PPS，里面包含了解码器的配置参数，比如profile、level、分辨率、帧率等.这些作为元数据存储在样本缓冲区的格式描述中，可以通过函数CMVideoFormatDescriptionGetH264ParameterSetAtIndex 访问.请注意，您必须在发送之前将开始代码添加到这些 NAL 单元中.SPS 和 PPS NAL 单元不必随每个新帧一起发送.解码器只需要读取它们一次，但通常会定期重新发送它们，例如在每个新的 I 帧 NAL 单元之前.
The final not very obvious thing is that the data inside the CMBlockBuffer does not contain the parameter NAL units SPS and PPS, which contains configuration parameters for the decoder such as profile, level, resolution, frame rate. These are stored as metadata in the sample buffer's format description and can be accessed via the function CMVideoFormatDescriptionGetH264ParameterSetAtIndex. Note that you have to add the start codes to these NAL units before sending. The SPS and PPS NAL units does not have to be sent with every new frame. A decoder only needs to read them once, but it is common to resend them periodically, for example before every new I-frame NAL unit.
下面是一个考虑到所有这些因素的代码示例.
Below is a code example taking all of these things into account.
static void videoFrameFinishedEncoding(void *outputCallbackRefCon,
void *sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer) {
// Check if there were any errors encoding
if (status != noErr) {
NSLog(@"Error encoding video, err=%lld", (int64_t)status);
return;
}
// In this example we will use a NSMutableData object to store the
// elementary stream.
NSMutableData *elementaryStream = [NSMutableData data];
// Find out if the sample buffer contains an I-Frame.
// If so we will write the SPS and PPS NAL units to the elementary stream.
BOOL isIFrame = NO;
CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
if (CFArrayGetCount(attachmentsArray)) {
CFBooleanRef notSync;
CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0);
BOOL keyExists = CFDictionaryGetValueIfPresent(dict,
kCMSampleAttachmentKey_NotSync,
(const void **)¬Sync);
// An I-Frame is a sync frame
isIFrame = !keyExists || !CFBooleanGetValue(notSync);
}
// This is the start code that we will write to
// the elementary stream before every NAL unit
static const size_t startCodeLength = 4;
static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01};
// Write the SPS and PPS NAL units to the elementary stream before every I-Frame
if (isIFrame) {
CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer);
// Find out how many parameter sets there are
size_t numberOfParameterSets;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
0, NULL, NULL,
&numberOfParameterSets,
NULL);
// Write each parameter set to the elementary stream
for (int i = 0; i < numberOfParameterSets; i++) {
const uint8_t *parameterSetPointer;
size_t parameterSetLength;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
i,
¶meterSetPointer,
¶meterSetLength,
NULL, NULL);
// Write the parameter set to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
[elementaryStream appendBytes:parameterSetPointer length:parameterSetLength];
}
}
// Get a pointer to the raw AVCC NAL unit data in the sample buffer
size_t blockBufferLength;
uint8_t *bufferDataPointer = NULL;
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer),
0,
NULL,
&blockBufferLength,
(char **)&bufferDataPointer);
// Loop through all the NAL units in the block buffer
// and write them to the elementary stream with
// start codes instead of AVCC length headers
size_t bufferOffset = 0;
static const int AVCCHeaderLength = 4;
while (bufferOffset < blockBufferLength - AVCCHeaderLength) {
// Read the NAL unit length
uint32_t NALUnitLength = 0;
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength);
// Convert the length value from Big-endian to Little-endian
NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
// Write start code to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
// Write the NAL unit without the AVCC length header to the elementary stream
[elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength
length:NALUnitLength];
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + NALUnitLength;
}
}
这篇关于从 CMBlockBuffer 中提取 h264的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
