在iPhone OpenGL ES 2.0中更快地替代glReadPixels [英] Faster alternative to glReadPixels in iPhone OpenGL ES 2.0

问题描述

有没有比使用glReadPixels更快的方式来访问帧缓冲区？我需要对帧缓冲区中的小矩形渲染区域进行只读访问，以便在CPU中进一步处理数据。性能很重要因为我必须重复执行此操作。我在网上搜索过并找到了一些方法，比如使用像素缓冲区对象和glMapBuffer，但似乎OpenGL ES 2.0不支持它们。

Is there any faster way to access the frame buffer than using glReadPixels? I would need read-only access to a small rectangular rendering area in the frame buffer to process the data further in CPU. Performance is important because I have to perform this operation repeatedly. I have searched the web and found some approach like using Pixel Buffer Object and glMapBuffer but it seems that OpenGL ES 2.0 does not support them.

推荐答案

从iOS 5.0开始，现在可以更快地从OpenGL ES中获取数据。它不是很明显，但事实证明，iOS 5.0中添加的纹理缓存支持不仅适用于将相机帧快速上传到OpenGL ES，而且可以反向使用以快速访问原始像素在OpenGL ES纹理中。

As of iOS 5.0, there is now a faster way to grab data from OpenGL ES. It isn't readily apparent, but it turns out that the texture cache support added in iOS 5.0 doesn't just work for fast upload of camera frames to OpenGL ES, but it can be used in reverse to get quick access to the raw pixels within an OpenGL ES texture.

您可以利用此功能通过使用带有附加纹理的帧缓冲对象（FBO）来获取OpenGL ES渲染的像素，纹理已从纹理缓存中提供。将场景渲染到FBO后，该场景的BGRA像素将包含在CVPixelBufferRef中，因此无需使用 glReadPixels（）将其拉下来。

You can take advantage of this to grab the pixels for an OpenGL ES rendering by using a framebuffer object (FBO) with an attached texture, with that texture having been supplied from the texture cache. Once you render your scene into that FBO, the BGRA pixels for that scene will be contained within your CVPixelBufferRef, so there will be no need to pull them down using glReadPixels().

这比在基准测试中使用 glReadPixels（）快得多。我发现在iPhone 4上， glReadPixels（）是读取720p视频帧以进行磁盘编码的瓶颈。它限制编码发生在超过8-9 FPS的任何地方。用快速纹理高速缓存读取代替它允许我现在以20 FPS编码720p视频，并且瓶颈已经从像素读取转移到OpenGL ES处理和管道的实际电影编码部分。在iPhone 4S上，这允许您以完整的30 FPS写入1080p视频。

This is much, much faster than using glReadPixels() in my benchmarks. I found that on my iPhone 4, glReadPixels() was the bottleneck in reading 720p video frames for encoding to disk. It limited the encoding from taking place at anything more than 8-9 FPS. Replacing this with the fast texture cache reads allows me to encode 720p video at 20 FPS now, and the bottleneck has moved from the pixel reading to the OpenGL ES processing and actual movie encoding parts of the pipeline. On an iPhone 4S, this allows you to write 1080p video at a full 30 FPS.

我的实现可以在我的开源中的GPUImageMovieWriter类中找到 GPUImage 框架，但它的灵感来自于 Dennis Muhlestein关于该主题的文章和Apple的ChromaKey示例应用程序（仅在WWDC上提供） 2011）。

My implementation can be found within the GPUImageMovieWriter class within my open source GPUImage framework, but it was inspired by Dennis Muhlestein's article on the subject and Apple's ChromaKey sample application (which was only made available at WWDC 2011).

我首先配置AVAssetWriter，添加输入，配置像素缓冲输入。以下代码用于设置像素缓冲区输入：

I start by configuring my AVAssetWriter, adding an input, and configuring a pixel buffer input. The following code is used to set up the pixel buffer input:

NSDictionary *sourcePixelBufferAttributesDictionary = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,
                                                       [NSNumber numberWithInt:videoSize.width], kCVPixelBufferWidthKey,
                                                       [NSNumber numberWithInt:videoSize.height], kCVPixelBufferHeightKey,
                                                       nil];

assetWriterPixelBufferInput = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:assetWriterVideoInput sourcePixelBufferAttributes:sourcePixelBufferAttributesDictionary];

我有了这个，我配置了我要渲染视频帧的FBO，使用以下代码：

Once I have that, I configure the FBO that I'll be rendering my video frames to, using the following code:

if ([GPUImageOpenGLESContext supportsFastTextureUpload])
{
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, (__bridge void *)[[GPUImageOpenGLESContext sharedImageProcessingOpenGLESContext] context], NULL, &coreVideoTextureCache);
    if (err) 
    {
        NSAssert(NO, @"Error at CVOpenGLESTextureCacheCreate %d");
    }

    CVPixelBufferPoolCreatePixelBuffer (NULL, [assetWriterPixelBufferInput pixelBufferPool], &renderTarget);

    CVOpenGLESTextureRef renderTexture;
    CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault, coreVideoTextureCache, renderTarget,
                                                  NULL, // texture attributes
                                                  GL_TEXTURE_2D,
                                                  GL_RGBA, // opengl format
                                                  (int)videoSize.width,
                                                  (int)videoSize.height,
                                                  GL_BGRA, // native iOS format
                                                  GL_UNSIGNED_BYTE,
                                                  0,
                                                  &renderTexture);

    glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0);
}

这从与我的资产编写者输入关联的池中提取像素缓冲区，创建并将纹理与其关联，并将该纹理用作我的FBO的目标。

This pulls a pixel buffer from the pool associated with my asset writer input, creates and associates a texture with it, and uses that texture as a target for my FBO.

一旦我渲染了一帧，我就锁定像素缓冲区的基地址：

Once I've rendered a frame, I lock the base address of the pixel buffer:

CVPixelBufferLockBaseAddress(pixel_buffer, 0);

然后只需将其输入我的资产编辑器进行编码：

and then simply feed it into my asset writer to be encoded:

CMTime currentTime = CMTimeMakeWithSeconds([[NSDate date] timeIntervalSinceDate:startTime],120);

if(![assetWriterPixelBufferInput appendPixelBuffer:pixel_buffer withPresentationTime:currentTime]) 
{
    NSLog(@"Problem appending pixel buffer at time: %lld", currentTime.value);
} 
else 
{
//        NSLog(@"Recorded pixel buffer at time: %lld", currentTime.value);
}
CVPixelBufferUnlockBaseAddress(pixel_buffer, 0);

if (![GPUImageOpenGLESContext supportsFastTextureUpload])
{
    CVPixelBufferRelease(pixel_buffer);
}

请注意，这里我没有手动阅读任何内容。此外，纹理本身采用BGRA格式，这是AVAssetWriters在编码视频时优化使用的，因此不需要在此处进行任何颜色调整。原始的BGRA像素只是被送入编码器来制作电影。

Note that at no point here am I reading anything manually. Also, the textures are natively in BGRA format, which is what AVAssetWriters are optimized to use when encoding video, so there's no need to do any color swizzling here. The raw BGRA pixels are just fed into the encoder to make the movie.

除了在AVAssetWriter中使用它之外，我在，我用于原始像素提取。与使用 glReadPixels（）相比，它在实践中也经历了显着的加速，尽管比我在AVAssetWriter中使用的像素缓冲池要少。

Aside from the use of this in an AVAssetWriter, I have some code in this answer that I've used for raw pixel extraction. It also experiences a significant speedup in practice when compared to using glReadPixels(), although less than I see with the pixel buffer pool I use with AVAssetWriter.

遗憾的是，这些都没有在任何地方记录，因为它大大提高了视频捕获性能。

It's a shame that none of this is documented anywhere, because it provides a huge boost to video capture performance.

这篇关于在iPhone OpenGL ES 2.0中更快地替代glReadPixels的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！