如何从 AVCaptureSession 产生的 CMSampleBuffer 中获取 Y 分量? [英] How to get the Y component from CMSampleBuffer resulted from the AVCaptureSession?

问题描述

嘿，我正在尝试使用 AVCaptureSession 从 iphone 相机访问原始数据.我遵循 Apple 提供的指南(此处链接).

Hey there, I am trying to access raw data from iphone camera using AVCaptureSession. I follow the guide provided by Apple (link here).

samplebuffer 中的原始数据是 YUV 格式的(我这里关于原始视频帧格式是否正确??)，如何从存储在 samplebuffer 中的原始数据中直接获取 Y 分量的数据.

The raw data from the samplebuffer is in YUV format ( Am I correct here about the raw video frame format?? ), how to directly obtain the data for Y component out of the raw data stored in the samplebuffer.

推荐答案

在设置返回原始相机帧的 AVCaptureVideoDataOutput 时，您可以使用如下代码设置帧的格式:

When setting up the AVCaptureVideoDataOutput that returns the raw camera frames, you can set the format of the frames using code like the following:

[videoOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];

在这种情况下，指定了 BGRA 像素格式(我用它来匹配 OpenGL ES 纹理的颜色格式).该格式中的每个像素都有一个字节，用于蓝色、绿色、红色和 alpha，按此顺序.使用此方法可以轻松提取颜色分量，但由于需要从相机原生 YUV 颜色空间进行转换，您确实会牺牲一点性能.

In this case a BGRA pixel format is specified (I used this for matching a color format for an OpenGL ES texture). Each pixel in that format has one byte for blue, green, red, and alpha, in that order. Going with this makes it easy to pull out color components, but you do sacrifice a little performance by needing to make the conversion from the camera-native YUV colorspace.

其他支持的颜色空间是 kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange 和 kCVPixelFormatType_420YpCbCr8BiPlanarFullRange(在较新的设备上)和 kCVPixelFormatType_422(iPhone 上的)VideoRange 或 FullRange 后缀仅指示返回的字节数是在 Y 的 16 - 235 和 UV 的 16 - 240 或每个组件的完整 0 - 255 之间.

Other supported colorspaces are kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange and kCVPixelFormatType_420YpCbCr8BiPlanarFullRange on newer devices and kCVPixelFormatType_422YpCbCr8 on the iPhone 3G. The VideoRange or FullRange suffix simply indicates whether the bytes are returned between 16 - 235 for Y and 16 - 240 for UV or full 0 - 255 for each component.

我相信 AVCaptureVideoDataOutput 实例使用的默认色彩空间是 YUV 4:2:0 平面色彩空间(除了在 iPhone 3G 上，它是 YUV 4:2:2 交错).这意味着视频帧中包含两个图像数据平面，Y 平面在前.对于结果图像中的每个像素，该像素的 Y 值都有一个字节.

I believe the default colorspace used by an AVCaptureVideoDataOutput instance is the YUV 4:2:0 planar colorspace (except on the iPhone 3G, where it's YUV 4:2:2 interleaved). This means that there are two planes of image data contained within the video frame, with the Y plane coming first. For every pixel in your resulting image, there is one byte for the Y value at that pixel.

您可以通过在委托回调中实现类似的东西来获取原始 Y 数据:

You would get at this raw Y data by implementing something like this in your delegate callback:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(pixelBuffer, 0);

    unsigned char *rawPixelBase = (unsigned char *)CVPixelBufferGetBaseAddress(pixelBuffer);

    // Do something with the raw pixels here

    CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
}

然后，您可以找出图像上每个 X、Y 坐标在帧数据中的位置，并拉出对应于该坐标处 Y 分量的字节.

You could then figure out the location in the frame data for each X, Y coordinate on the image and pull the byte out that corresponds to the Y component at that coordinate.

来自 WWDC 2010 的 Apple FindMyiCone 示例(可与视频一起访问)展示了如何处理来自每一帧的原始 BGRA 数据.我还创建了一个示例应用程序，您可以在此处下载代码，执行基于颜色的对象跟踪 使用来自 iPhone 相机的实时视频.两者都展示了如何处理原始像素数据，但它们都不适用于 YUV 色彩空间.

Apple's FindMyiCone sample from WWDC 2010 (accessible along with the videos) shows how to process raw BGRA data from each frame. I also created a sample application, which you can download the code for here, that performs color-based object tracking using the live video from the iPhone's camera. Both show how to process raw pixel data, but neither of these work in the YUV colorspace.

这篇关于如何从 AVCaptureSession 产生的 CMSampleBuffer 中获取 Y 分量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！