iOS CVImageBuffer 从 AVCaptureSessionDataOutput 失真，带有 AVCaptureSessionPresetPhoto [英] iOS CVImageBuffer distorted from AVCaptureSessionDataOutput with AVCaptureSessionPresetPhoto
问题描述
在高层次上，我创建了一个应用程序，让用户可以将他或她的 iPhone 摄像头指向四周，并查看经过视觉效果处理的视频帧.此外，用户可以点击一个按钮，将当前预览的冻结帧作为高分辨率照片保存在他们的 iPhone 库中.
At a high level, I created an app that lets a user point his or her iPhone camera around and see video frames that have been processed with visual effects. Additionally, the user can tap a button to take a freeze-frame of the current preview as a high-resolution photo that is saved in their iPhone library.
为此，应用程序遵循以下程序:
To do this, the app follows this procedure:
1) 创建一个 AVCaptureSession
1) Create an AVCaptureSession
captureSession = [[AVCaptureSession alloc] init];
[captureSession setSessionPreset:AVCaptureSessionPreset640x480];
2) 使用后置摄像头连接 AVCaptureDeviceInput.
2) Hook up an AVCaptureDeviceInput using the back-facing camera.
videoInput = [[[AVCaptureDeviceInput alloc] initWithDevice:backFacingCamera error:&error] autorelease];
[captureSession addInput:videoInput];
3) 将 AVCaptureStillImageOutput 连接到会话，以便能够以照片分辨率捕获静止帧.
3) Hook up an AVCaptureStillImageOutput to the session to be able to capture still frames at Photo resolution.
stillOutput = [[AVCaptureStillImageOutput alloc] init];
[stillOutput setOutputSettings:[NSDictionary
dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA]
forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
[captureSession addOutput:stillOutput];
4) 将 AVCaptureVideoDataOutput 连接到会话，以便能够以较低的分辨率捕获单个视频帧 (CVImageBuffers)
4) Hook up an AVCaptureVideoDataOutput to the session to be able to capture individual video frames (CVImageBuffers) at a lower resolution
videoOutput = [[AVCaptureVideoDataOutput alloc] init];
[videoOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
[videoOutput setSampleBufferDelegate:self queue:dispatch_get_main_queue()];
[captureSession addOutput:videoOutput];
5) 当视频帧被捕获时，委托的方法被调用，每个新帧作为一个 CVImageBuffer:
5) As video frames are captured, the delegate's method is called with each new frame as a CVImageBuffer:
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
[self.delegate processNewCameraFrame:pixelBuffer];
}
6) 然后委托处理/绘制它们:
6) Then the delegate processes/draws them:
- (void)processNewCameraFrame:(CVImageBufferRef)cameraFrame {
CVPixelBufferLockBaseAddress(cameraFrame, 0);
int bufferHeight = CVPixelBufferGetHeight(cameraFrame);
int bufferWidth = CVPixelBufferGetWidth(cameraFrame);
glClear(GL_COLOR_BUFFER_BIT);
glGenTextures(1, &videoFrameTexture_);
glBindTexture(GL_TEXTURE_2D, videoFrameTexture_);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bufferWidth, bufferHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress(cameraFrame));
glBindBuffer(GL_ARRAY_BUFFER, [self vertexBuffer]);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, [self indexBuffer]);
glDrawElements(GL_TRIANGLE_STRIP, 4, GL_UNSIGNED_SHORT, BUFFER_OFFSET(0));
glBindBuffer(GL_ARRAY_BUFFER, 0);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, 0);
[[self context] presentRenderbuffer:GL_RENDERBUFFER];
glDeleteTextures(1, &videoFrameTexture_);
CVPixelBufferUnlockBaseAddress(cameraFrame, 0);
}
这一切都有效并导致正确的结果.我可以看到通过 OpenGL 处理的 640x480 的视频预览.它看起来像这样:
This all works and leads to the correct results. I can see a video preview of 640x480 processed through OpenGL. It looks like this:
但是，如果我从该会话中捕获静止图像，其分辨率也将是 640x480.我希望它是高分辨率的，所以在第一步中我将预设行更改为:
However, if I capture a still image from this session, its resolution will also be 640x480. I want it to be high resolution, so in step one I change the preset line to:
[captureSession setSessionPreset:AVCaptureSessionPresetPhoto];
这可以正确捕捉 iPhone4 的最高分辨率 (2592x1936) 的静止图像.
This correctly captures still images at the highest resolution for the iPhone4 (2592x1936).
但是，视频预览(由代表在第 5 步和第 6 步中收到)现在看起来像这样:
However, the video preview (as received by the delegate in steps 5 and 6) now looks like this:
我已确认所有其他预设(高、中、低、640x480 和 1280x720)都能按预期进行预览.但是，照片预设似乎以不同的格式发送缓冲区数据.
I've confirmed that every other preset (High, medium, low, 640x480, and 1280x720) previews as intended. However, the Photo preset seems to send buffer data in a different format.
我还通过获取缓冲区并从中创建 UIImage 而不是将其发送到 openGL 来确认发送到照片预设缓冲区的数据实际上是有效的图像数据:
I've also confirmed that the data being sent to the buffer at the Photo preset is actually valid image data by taking the buffer and creating a UIImage out of it instead of sending it to openGL:
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(CVPixelBufferGetBaseAddress(cameraFrame), bufferWidth, bufferHeight, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGImageRef cgImage = CGBitmapContextCreateImage(context);
UIImage *anImage = [UIImage imageWithCGImage:cgImage];
这显示了一个未失真的视频帧.
This shows an undistorted video frame.
我进行了大量搜索，但似乎无法修复它.我的直觉是这是一个数据格式问题.也就是说，我相信缓冲区设置正确，但格式是这一行无法理解的:
I've done a bunch of searching and can't seem to fix it. My hunch is that it's a data format issue. That is, I believe that the buffer is being set correctly, but with a format that this line doesn't understand:
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bufferWidth, bufferHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress(cameraFrame));
我的预感是，将外部格式从 GL_BGRA 更改为其他格式会有所帮助，但它并没有……通过各种方式，看起来缓冲区实际上位于 GL_BGRA 中.
My hunch was that changing the external format from GL_BGRA to something else would help, but it doesn't... and through various means it looks like the buffer is actually in GL_BGRA.
有人知道这里发生了什么吗?或者您对我如何调试为什么会发生这种情况有任何提示?(超级奇怪的是，这发生在 iphone4 而不是 iPhone 3GS 上……都运行 ios4.3)
Does anyone know what's going on here? Or do you have any tips on how I might go about debugging why this is happening? (What's super weird is that this happens on an iphone4 but not on an iPhone 3GS ... both running ios4.3)
推荐答案
这太棒了.
正如 Lio Ben-Kereth 指出的那样，从调试器中可以看到，填充是 48
As Lio Ben-Kereth pointed out, the padding is 48 as you can see from the debugger
(gdb) po pixelBuffer
<CVPixelBuffer 0x2934d0 width=852 height=640 bytesPerRow=3456 pixelFormat=BGRA
# => 3456 - 852 * 4 = 48
OpenGL 可以弥补这一点，但 OpenGL ES 不能(更多信息请点击此处openGL SubTexturinga>)
OpenGL can compensate for this, but OpenGL ES cannot (more info here openGL SubTexturing)
这就是我在 OpenGL ES 中的做法:
So here is how I'm doing it in OpenGL ES:
(CVImageBufferRef)pixelBuffer // pixelBuffer containing the raw image data is passed in
/* ... */
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, videoFrameTexture_);
int frameWidth = CVPixelBufferGetWidth(pixelBuffer);
int frameHeight = CVPixelBufferGetHeight(pixelBuffer);
size_t bytesPerRow, extraBytes;
bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
extraBytes = bytesPerRow - frameWidth*4;
GLubyte *pixelBufferAddr = CVPixelBufferGetBaseAddress(pixelBuffer);
if ( [[captureSession sessionPreset] isEqualToString:@"AVCaptureSessionPresetPhoto"] )
{
glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, frameWidth, frameHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, NULL );
for( int h = 0; h < frameHeight; h++ )
{
GLubyte *row = pixelBufferAddr + h * (frameWidth * 4 + extraBytes);
glTexSubImage2D( GL_TEXTURE_2D, 0, 0, h, frameWidth, 1, GL_BGRA, GL_UNSIGNED_BYTE, row );
}
}
else
{
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, frameWidth, frameHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, pixelBufferAddr);
}
之前，我使用 AVCaptureSessionPresetMedium 并获得 30fps.在 AVCaptureSessionPresetPhoto 中，我在 iPhone 4 上获得了 16fps.子纹理的循环似乎不会影响帧速率.
Before, I was using AVCaptureSessionPresetMedium and getting 30fps. In AVCaptureSessionPresetPhoto I'm getting 16fps on an iPhone 4. The looping for the sub-texture does not seem to affect the frame rate.
我在 iOS 5 上使用 iPhone 4.
I'm using an iPhone 4 on iOS 5.
这篇关于iOS CVImageBuffer 从 AVCaptureSessionDataOutput 失真，带有 AVCaptureSessionPresetPhoto的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
