iOS CVImageBuffer使用AVCaptureSessionPresetPhoto从AVCaptureSessionDataOutput失真 [英] iOS CVImageBuffer distorted from AVCaptureSessionDataOutput with AVCaptureSessionPresetPhoto

问题描述

在最高级别，我创建了一个应用程序，让用户可以指向他或她的iPhone相机，并查看已使用视觉效果处理过的视频帧。此外，用户可以点击按钮将当前预览的定格作为保存在iPhone库中的高分辨率照片。

At a high level, I created an app that lets a user point his or her iPhone camera around and see video frames that have been processed with visual effects. Additionally, the user can tap a button to take a freeze-frame of the current preview as a high-resolution photo that is saved in their iPhone library.

要执行此操作，应用程序遵循以下程序：

To do this, the app follows this procedure:

1）创建AVCaptureSession

1) Create an AVCaptureSession

captureSession = [[AVCaptureSession alloc] init];
[captureSession setSessionPreset:AVCaptureSessionPreset640x480];

2）使用后置摄像头连接AVCaptureDeviceInput。

2) Hook up an AVCaptureDeviceInput using the back-facing camera.

videoInput = [[[AVCaptureDeviceInput alloc] initWithDevice:backFacingCamera error:&error] autorelease];
[captureSession addInput:videoInput];

3）将AVCaptureStillImageOutput连接到会话，以便能够以Photo分辨率捕获静止帧。 / p>

3) Hook up an AVCaptureStillImageOutput to the session to be able to capture still frames at Photo resolution.

stillOutput = [[AVCaptureStillImageOutput alloc] init];
[stillOutput setOutputSettings:[NSDictionary
    dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA]
    forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
[captureSession addOutput:stillOutput];

4）将AVCaptureVideoDataOutput连接到会话，以便能够捕获单个视频帧（CVImageBuffers）at分辨率较低

4) Hook up an AVCaptureVideoDataOutput to the session to be able to capture individual video frames (CVImageBuffers) at a lower resolution

videoOutput = [[AVCaptureVideoDataOutput alloc] init];
[videoOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
[videoOutput setSampleBufferDelegate:self queue:dispatch_get_main_queue()];
[captureSession addOutput:videoOutput];

5）在捕获视频帧时，调用委托的方法，每个新帧作为CVImageBuffer：

5) As video frames are captured, the delegate's method is called with each new frame as a CVImageBuffer:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    [self.delegate processNewCameraFrame:pixelBuffer];
}

6）然后委托处理/绘制它们：

6) Then the delegate processes/draws them:

- (void)processNewCameraFrame:(CVImageBufferRef)cameraFrame {
    CVPixelBufferLockBaseAddress(cameraFrame, 0);
    int bufferHeight = CVPixelBufferGetHeight(cameraFrame);
    int bufferWidth = CVPixelBufferGetWidth(cameraFrame);

    glClear(GL_COLOR_BUFFER_BIT);

    glGenTextures(1, &videoFrameTexture_);
    glBindTexture(GL_TEXTURE_2D, videoFrameTexture_);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bufferWidth, bufferHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress(cameraFrame));

    glBindBuffer(GL_ARRAY_BUFFER, [self vertexBuffer]);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, [self indexBuffer]);

    glDrawElements(GL_TRIANGLE_STRIP, 4, GL_UNSIGNED_SHORT, BUFFER_OFFSET(0));

    glBindBuffer(GL_ARRAY_BUFFER, 0);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, 0);
    [[self context] presentRenderbuffer:GL_RENDERBUFFER];

    glDeleteTextures(1, &videoFrameTexture_);

    CVPixelBufferUnlockBaseAddress(cameraFrame, 0);
}

这一切都有效并导致正确的结果。我可以通过OpenGL看到640x480的视频预览。它看起来像这样：

This all works and leads to the correct results. I can see a video preview of 640x480 processed through OpenGL. It looks like this:

但是，如果我从此会话中捕获静止图像，其分辨率也将为640x480。我希望它是高分辨率，所以在第一步中我将预设线更改为：

However, if I capture a still image from this session, its resolution will also be 640x480. I want it to be high resolution, so in step one I change the preset line to:

[captureSession setSessionPreset:AVCaptureSessionPresetPhoto];

这可正确捕捉iPhone4（2592x1936）最高分辨率的静止图像。

This correctly captures still images at the highest resolution for the iPhone4 (2592x1936).

然而，视频预览（由代表在步骤5和6中收到）现在看起来像这样：

However, the video preview (as received by the delegate in steps 5 and 6) now looks like this:

我已确认所有其他预设（高，中，低，640x480和1280x720）预览。但是，Photo预设似乎以不同的格式发送缓冲区数据。

I've confirmed that every other preset (High, medium, low, 640x480, and 1280x720) previews as intended. However, the Photo preset seems to send buffer data in a different format.

我还确认在Photo预设中发送到缓冲区的数据实际上是有效的图像数据通过获取缓冲区并从中创建UIImage而不是将其发送到openGL：

I've also confirmed that the data being sent to the buffer at the Photo preset is actually valid image data by taking the buffer and creating a UIImage out of it instead of sending it to openGL:

CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(CVPixelBufferGetBaseAddress(cameraFrame), bufferWidth, bufferHeight, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst); 
CGImageRef cgImage = CGBitmapContextCreateImage(context); 
UIImage *anImage = [UIImage imageWithCGImage:cgImage];

这显示未失真的视频帧。

This shows an undistorted video frame.

我做了一堆搜索，似乎无法修复它。我的预感是这是一个数据格式问题。也就是说，我相信缓冲区设置正确，但是这行格式不明白：

I've done a bunch of searching and can't seem to fix it. My hunch is that it's a data format issue. That is, I believe that the buffer is being set correctly, but with a format that this line doesn't understand:

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bufferWidth, bufferHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress(cameraFrame));

我的预感是将外部格式从GL_BGRA更改为其他内容会有所帮助，但它不会......并且通过各种方式看起来缓冲区实际上在GL_BGRA中。

My hunch was that changing the external format from GL_BGRA to something else would help, but it doesn't... and through various means it looks like the buffer is actually in GL_BGRA.

有谁知道这里发生了什么？或者你有任何关于如何调试为什么会这样做的提示？ （超级奇怪的是，这种情况发生在iphone4上，而不是iPhone 3GS上......运行ios4.3）

Does anyone know what's going on here? Or do you have any tips on how I might go about debugging why this is happening? (What's super weird is that this happens on an iphone4 but not on an iPhone 3GS ... both running ios4.3)

推荐答案

这很麻烦。

正如Lio Ben-Kereth指出的那样，从调试器中可以看出填充是48

As Lio Ben-Kereth pointed out, the padding is 48 as you can see from the debugger

(gdb) po pixelBuffer
<CVPixelBuffer 0x2934d0 width=852 height=640 bytesPerRow=3456 pixelFormat=BGRA
# => 3456 - 852 * 4 = 48

OpenGL可以弥补这一点，但OpenGL ES 不能（更多信息请 openGL SubTexturing ）

OpenGL can compensate for this, but OpenGL ES cannot (more info here openGL SubTexturing)

所以这就是我在OpenGL ES中的表现：

So here is how I'm doing it in OpenGL ES:

(CVImageBufferRef)pixelBuffer   // pixelBuffer containing the raw image data is passed in

/* ... */
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, videoFrameTexture_);

int frameWidth = CVPixelBufferGetWidth(pixelBuffer);
int frameHeight = CVPixelBufferGetHeight(pixelBuffer);

size_t bytesPerRow, extraBytes;

bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
extraBytes = bytesPerRow - frameWidth*4;

GLubyte *pixelBufferAddr = CVPixelBufferGetBaseAddress(pixelBuffer);

if ( [[captureSession sessionPreset] isEqualToString:@"AVCaptureSessionPresetPhoto"] )
{

    glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, frameWidth, frameHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, NULL );

    for( int h = 0; h < frameHeight; h++ )
    {
        GLubyte *row = pixelBufferAddr + h * (frameWidth * 4 + extraBytes);
        glTexSubImage2D( GL_TEXTURE_2D, 0, 0, h, frameWidth, 1, GL_BGRA, GL_UNSIGNED_BYTE, row );
    }
}
else
{
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, frameWidth, frameHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, pixelBufferAddr);
}

之前，我使用 AVCaptureSessionPresetMedium 并获得30fps。在 AVCaptureSessionPresetPhoto 我在iPhone 4上获得16fps。子纹理的循环似乎不会影响帧速率。

Before, I was using AVCaptureSessionPresetMedium and getting 30fps. In AVCaptureSessionPresetPhoto I'm getting 16fps on an iPhone 4. The looping for the sub-texture does not seem to affect the frame rate.

我在iOS 5上使用的是iPhone 4。

I'm using an iPhone 4 on iOS 5.

这篇关于iOS CVImageBuffer使用AVCaptureSessionPresetPhoto从AVCaptureSessionDataOutput失真的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！