从CMBlockBuffer提取h264 [英] Extracting h264 from CMBlockBuffer
问题描述
我使用Apple VideoTool Box（iOS）来压缩由设备相机捕获的原始帧。
I am using Apple VideoTool Box (iOS) to compress raw frames captured by the device camera.
我的回调被包含CMBlockBuffer的CMSampleBufferRef对象调用。
My callback is being called with a CMSampleBufferRef object that contains CMBlockBuffer.
CMBlockBuffer对象包含H264基本流，但我没有找到任何方法来获取基本流的指针。
The CMBlockBuffer object contain the H264 elementary stream but I didn't find any way to get a pointer to the elementary stream.
当我打印到控制台CMSampleBufferRef对象我得到：
When I printed into the console the CMSampleBufferRef object I got:
(lldb) po blockBufferRef
CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4264 bytes @ offset 128 Buffer Reference:
CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
[0] 4632 bytes @ offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0)
任何人都可以发布如何访问H264元素流？
Can anyone post how to access the H264 elemantry stream?
谢谢！
推荐答案
我一直在努力
函数 CMBlockBufferGetDataPointer 使您可以访问所有的
The function CMBlockBufferGetDataPointer gives you access to all the data you need, but there are a few not very obvious things you need to do to convert it to an elementary stream.
CMBlockBuffer中的数据以AVCC格式存储，而基本流通常遵循附件B规范（这里是两种格式的一个很好的概述）。在AVCC格式中，4个第一字节包含NAL单元的长度（H264分组的另一个字）。您需要用4字节的起始码替换此标头：0x00 0x00 0x00 0x01，它用作附件B基本流中的NAL单元之间的分隔符（3字节版本0x00 0x00 0x01也正常工作）。
The data in the CMBlockBuffer is stored in AVCC format, while elementary streams are typically following the Annex B specification (here is an excellent overview of the two formats). In the AVCC format, the 4 first bytes contains the length of the NAL unit (another word for H264 packet). You need to replace this header with the 4 byte start code: 0x00 0x00 0x00 0x01, which functions as a separator between NAL units in an Annex B elementary stream (the 3 byte version 0x00 0x00 0x01 works fine too).
下一个不明显的事情是，单个CMBlockBuffer有时会包含多个NAL单元。苹果似乎向每个I帧NAL单元（也称为IDR）添加包含元数据的附加NAL单元（SEI）。这可能是为什么你在一个CMBlockBuffer对象中看到多个缓冲区。但是， CMBlockBufferGetDataPointer 函数为您提供了访问所有数据的单个指针。话虽如此，多个NAL单元的存在使AVCC头部的转换复杂化。现在你实际上必须读取AVCC头中包含的长度值，以找到下一个NAL单元，并继续转换头，直到到达缓冲区的末尾。
The next not very obvious thing is that a single CMBlockBuffer will sometimes contain multiple NAL units. Apple seems to add an additional NAL unit (SEI) containing metadata to every I-Frame NAL unit (also called IDR). This is probably why you are seeing multiple buffers in a single CMBlockBuffer object. However, the CMBlockBufferGetDataPointer function gives you a single pointer with access to all the data. That being said, the presence of multiple NAL units complicates the conversion of the AVCC headers. Now you actually have to read the length value contained in the AVCC header to find the next NAL unit, and continue converting headers until you have reached the end of the buffer.
下一个不是很明显的事情是，AVCC头是以Big-Endian格式存储的，iOS是本地的Little-Endian。因此，当您读取AVCC头中包含的长度值时，首先将其传递给 CFSwapInt32BigToHost 函数。
The next not very obvious thing is that the AVCC header is stored in Big-Endian format, and iOS is Little-Endian natively. So when you are reading the length value contained in an AVCC header pass it to the CFSwapInt32BigToHost function first.
最后一个不是很明显的事情是，CMBlockBuffer中的数据不包含参数NAL单元SPS和PPS，其包含解码器的配置参数例如配置文件，级别，分辨率，帧速率。这些作为元数据存储在样本缓冲区的格式描述中，并且可以通过 CMVideoFormatDescriptionGetH264ParameterSetAtIndex 函数访问。注意，您必须在发送之前将起始码添加到这些NAL单元。 SPS和PPS NAL单元不必与每个新帧一起发送。解码器只需要读取它们一次，但是通常周期性地重新发送它们，例如在每个新的I帧NAL单元之前。
The final not very obvious thing is that the data inside the CMBlockBuffer does not contain the parameter NAL units SPS and PPS, which contains configuration parameters for the decoder such as profile, level, resolution, frame rate. These are stored as metadata in the sample buffer's format description and can be accessed via the function CMVideoFormatDescriptionGetH264ParameterSetAtIndex. Note that you have to add the start codes to these NAL units before sending. The SPS and PPS NAL units does not have to be sent with every new frame. A decoder only needs to read them once, but it is common to resend them periodically, for example before every new I-frame NAL unit.
下面是一个考虑所有这些事情的代码示例。
Below is a code example taking all of these things into account.
static void videoFrameFinishedEncoding(void *outputCallbackRefCon,
void *sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer) {
// Check if there were any errors encoding
if (status != noErr) {
NSLog(@"Error encoding video, err=%lld", (int64_t)status);
return;
}
// In this example we will use a NSMutableData object to store the
// elementary stream.
NSMutableData *elementaryStream = [NSMutableData data];
// Find out if the sample buffer contains an I-Frame.
// If so we will write the SPS and PPS NAL units to the elementary stream.
BOOL isIFrame = NO;
CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
if (CFArrayGetCount(attachmentsArray)) {
CFBooleanRef notSync;
CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0);
BOOL keyExists = CFDictionaryGetValueIfPresent(dict,
kCMSampleAttachmentKey_NotSync,
(const void **)¬Sync);
// An I-Frame is a sync frame
isIFrame = !keyExists || !CFBooleanGetValue(notSync);
}
// This is the start code that we will write to
// the elementary stream before every NAL unit
static const size_t startCodeLength = 4;
static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01};
// Write the SPS and PPS NAL units to the elementary stream before every I-Frame
if (isIFrame) {
CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer);
// Find out how many parameter sets there are
size_t numberOfParameterSets;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
0, NULL, NULL,
&numberOfParameterSets,
NULL);
// Write each parameter set to the elementary stream
for (int i = 0; i < numberOfParameterSets; i++) {
const uint8_t *parameterSetPointer;
size_t parameterSetLength;
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
i,
¶meterSetPointer,
¶meterSetLength,
NULL, NULL);
// Write the parameter set to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
[elementaryStream appendBytes:parameterSetPointer length:parameterSetLength];
}
}
// Get a pointer to the raw AVCC NAL unit data in the sample buffer
size_t blockBufferLength;
uint8_t *bufferDataPointer = NULL;
CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer),
0,
NULL,
&blockBufferLength,
(char **)&bufferDataPointer);
// Loop through all the NAL units in the block buffer
// and write them to the elementary stream with
// start codes instead of AVCC length headers
size_t bufferOffset = 0;
static const int AVCCHeaderLength = 4;
while (bufferOffset < blockBufferLength - AVCCHeaderLength) {
// Read the NAL unit length
uint32_t NALUnitLength = 0;
memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength);
// Convert the length value from Big-endian to Little-endian
NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
// Write start code to the elementary stream
[elementaryStream appendBytes:startCode length:startCodeLength];
// Write the NAL unit without the AVCC length header to the elementary stream
[elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength
length:NALUnitLength];
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + NALUnitLength;
}
}
这篇关于从CMBlockBuffer提取h264的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
