iPhone OpenGL ES 2.0 中 glReadPixels 的更快替代方案 [英] Faster alternative to glReadPixels in iPhone OpenGL ES 2.0

问题描述

有没有比使用 glReadPixels 访问帧缓冲区更快的方法?我需要对帧缓冲区中的一个小的矩形渲染区域进行只读访问，以便在 CPU 中进一步处理数据.性能很重要，因为我必须重复执行此操作.我在网上搜索并找到了一些方法，例如使用 Pixel Buffer Object 和 glMapBuffer 但似乎 OpenGL ES 2.0 不支持它们.

Is there any faster way to access the frame buffer than using glReadPixels? I would need read-only access to a small rectangular rendering area in the frame buffer to process the data further in CPU. Performance is important because I have to perform this operation repeatedly. I have searched the web and found some approach like using Pixel Buffer Object and glMapBuffer but it seems that OpenGL ES 2.0 does not support them.

推荐答案

从 iOS 5.0 开始，现在可以更快地从 OpenGL ES 获取数据.这不是很明显，但事实证明，iOS 5.0 中添加的纹理缓存支持不仅适用于将相机帧快速上传到 OpenGL ES，而且可以反向使用以快速访问原始像素在 OpenGL ES 纹理中.

As of iOS 5.0, there is now a faster way to grab data from OpenGL ES. It isn't readily apparent, but it turns out that the texture cache support added in iOS 5.0 doesn't just work for fast upload of camera frames to OpenGL ES, but it can be used in reverse to get quick access to the raw pixels within an OpenGL ES texture.

您可以利用这一点，通过使用带有附加纹理的帧缓冲区对象 (FBO) 来获取 OpenGL ES 渲染的像素，该纹理是从纹理缓存提供的.一旦您将场景渲染到该 FBO 中，该场景的 BGRA 像素将包含在您的 CVPixelBufferRef 中，因此无需使用 glReadPixels() 将它们拉下来.

You can take advantage of this to grab the pixels for an OpenGL ES rendering by using a framebuffer object (FBO) with an attached texture, with that texture having been supplied from the texture cache. Once you render your scene into that FBO, the BGRA pixels for that scene will be contained within your CVPixelBufferRef, so there will be no need to pull them down using glReadPixels().

这比在我的基准测试中使用 glReadPixels() 快得多.我发现在我的 iPhone 4 上，glReadPixels() 是读取 720p 视频帧以编码到磁盘的瓶颈.它限制了编码以超过 8-9 FPS 的速度发生.用快速纹理缓存读取代替它，我现在可以以 20 FPS 编码 720p 视频，瓶颈已经从像素读取转移到 OpenGL ES 处理和管道的实际电影编码部分.在 iPhone 4S 上，这允许您以完整的 30 FPS 写入 1080p 视频.

This is much, much faster than using glReadPixels() in my benchmarks. I found that on my iPhone 4, glReadPixels() was the bottleneck in reading 720p video frames for encoding to disk. It limited the encoding from taking place at anything more than 8-9 FPS. Replacing this with the fast texture cache reads allows me to encode 720p video at 20 FPS now, and the bottleneck has moved from the pixel reading to the OpenGL ES processing and actual movie encoding parts of the pipeline. On an iPhone 4S, this allows you to write 1080p video at a full 30 FPS.

我的实现可以在我的开源 GPUImage 框架的 GPUImageMovieWriter 类中找到，但它受到启发来自 Dennis Muhlestein 的文章 和 Apple 的 ChromaKey 示例应用程序(仅在 WWDC 2011 上提供).

My implementation can be found within the GPUImageMovieWriter class within my open source GPUImage framework, but it was inspired by Dennis Muhlestein's article on the subject and Apple's ChromaKey sample application (which was only made available at WWDC 2011).

我首先配置我的 AVAssetWriter，添加一个输入，然后配置一个像素缓冲区输入.以下代码用于设置像素缓冲区输入:

I start by configuring my AVAssetWriter, adding an input, and configuring a pixel buffer input. The following code is used to set up the pixel buffer input:

NSDictionary *sourcePixelBufferAttributesDictionary = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,
                                                       [NSNumber numberWithInt:videoSize.width], kCVPixelBufferWidthKey,
                                                       [NSNumber numberWithInt:videoSize.height], kCVPixelBufferHeightKey,
                                                       nil];

assetWriterPixelBufferInput = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:assetWriterVideoInput sourcePixelBufferAttributes:sourcePixelBufferAttributesDictionary];

一旦我有了它，我就会使用以下代码配置我要将视频帧渲染到的 FBO:

Once I have that, I configure the FBO that I'll be rendering my video frames to, using the following code:

if ([GPUImageOpenGLESContext supportsFastTextureUpload])
{
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, (__bridge void *)[[GPUImageOpenGLESContext sharedImageProcessingOpenGLESContext] context], NULL, &coreVideoTextureCache);
    if (err) 
    {
        NSAssert(NO, @"Error at CVOpenGLESTextureCacheCreate %d");
    }

    CVPixelBufferPoolCreatePixelBuffer (NULL, [assetWriterPixelBufferInput pixelBufferPool], &renderTarget);

    CVOpenGLESTextureRef renderTexture;
    CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault, coreVideoTextureCache, renderTarget,
                                                  NULL, // texture attributes
                                                  GL_TEXTURE_2D,
                                                  GL_RGBA, // opengl format
                                                  (int)videoSize.width,
                                                  (int)videoSize.height,
                                                  GL_BGRA, // native iOS format
                                                  GL_UNSIGNED_BYTE,
                                                  0,
                                                  &renderTexture);

    glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0);
}

这会从与我的资产编写器输入关联的池中提取一个像素缓冲区，创建纹理并将其与其关联，然后将该纹理用作我的 FBO 的目标.

This pulls a pixel buffer from the pool associated with my asset writer input, creates and associates a texture with it, and uses that texture as a target for my FBO.

一旦我渲染了一帧，我就会锁定像素缓冲区的基地址:

Once I've rendered a frame, I lock the base address of the pixel buffer:

CVPixelBufferLockBaseAddress(pixel_buffer, 0);

然后简单地将其输入到我的资产编写器中进行编码:

and then simply feed it into my asset writer to be encoded:

CMTime currentTime = CMTimeMakeWithSeconds([[NSDate date] timeIntervalSinceDate:startTime],120);

if(![assetWriterPixelBufferInput appendPixelBuffer:pixel_buffer withPresentationTime:currentTime]) 
{
    NSLog(@"Problem appending pixel buffer at time: %lld", currentTime.value);
} 
else 
{
//        NSLog(@"Recorded pixel buffer at time: %lld", currentTime.value);
}
CVPixelBufferUnlockBaseAddress(pixel_buffer, 0);

if (![GPUImageOpenGLESContext supportsFastTextureUpload])
{
    CVPixelBufferRelease(pixel_buffer);
}

请注意，我从来没有手动阅读任何内容.此外，纹理本身采用 BGRA 格式，这是 AVAssetWriters 在编码视频时优化使用的格式，因此无需在此处进行任何颜色调整.原始 BGRA 像素仅被输入编码器以制作电影.

Note that at no point here am I reading anything manually. Also, the textures are natively in BGRA format, which is what AVAssetWriters are optimized to use when encoding video, so there's no need to do any color swizzling here. The raw BGRA pixels are just fed into the encoder to make the movie.

除了在 AVAssetWriter 中使用 this 之外，我在 this answer 中有一些代码，我已经用于原始像素提取.与使用 glReadPixels() 相比，它在实践中也经历了显着的加速，尽管比我在 AVAssetWriter 使用的像素缓冲池中看到的要少.

Aside from the use of this in an AVAssetWriter, I have some code in this answer that I've used for raw pixel extraction. It also experiences a significant speedup in practice when compared to using glReadPixels(), although less than I see with the pixel buffer pool I use with AVAssetWriter.

遗憾的是，这一切都没有记录在任何地方，因为它极大地提高了视频捕获性能.

It's a shame that none of this is documented anywhere, because it provides a huge boost to video capture performance.

这篇关于iPhone OpenGL ES 2.0 中 glReadPixels 的更快替代方案的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！