将RGB像素数据高性能复制到iOS中的屏幕 [英] High-performance copying of RGB pixel data to the screen in iOS

问题描述

我们的产品包含一种软件图像解码器，该解码器本质上会产生需要快速复制到屏幕上的全帧像素数据(我们在iOS上运行).

Our product contains a kind of software image decoder that essentially produces full-frame pixel data that needs to be rapidly copied the screen (we're running on iOS).

当前，我们正在使用CGBitmapContextCreate，我们直接访问内存缓冲区，然后对于每个帧，我们调用CGBitmapContextCreateImage，然后将该位图绘制到屏幕上.对于以适当的帧速率在iPad的视网膜显示屏上进行全屏刷新来说，这太慢了(但是对于非Retina设备来说也可以).

Currently we're using CGBitmapContextCreate and we access the memory buffer directly, then for each frame we call CGBitmapContextCreateImage, and then draw that bitmap to the screen. This is WAY too slow for full-screen refreshes on the iPad's retina display at a decent framerate (but it was okay for non-Retina-devices).

我们已经尝试了各种基于OpenGL ES的方法，包括使用glTexImage2D和glTexSubImage2D(本质上是渲染到纹理)，但是CPU使用率仍然很高，并且完全不能超过30 FPS屏幕在iPad 3上刷新.问题在于，以30 FPS的速度，CPU使用率几乎为％100，仅用于将像素复制到屏幕上，这意味着我们在CPU上进行渲染时不需要太多工作

We've tried all kinds of OpenGL ES-based approaches, including the use of glTexImage2D and glTexSubImage2D (essentially rendering to a texture), but CPU usage is still high and we can't get more than ~30 FPS for full-screen refreshes on the iPad 3. The problem is that with 30 FPS, CPU usage is nearly at %100 just for copying the pixels to the screen, which means we don't have much to work with for our own rendering on the CPU.

我们愿意使用OpenGL或任何能给我们带来最佳性能的iOS API.像素数据被格式化为每像素32位RGBA数据，但是我们在其中具有一定的灵活性...

We are open to using OpenGL or any iOS API that would give us maximum performance. The pixel data is formatted as a 32-bit-per-pixel RGBA data but we have some flexibility there...

有什么建议吗?

推荐答案

因此，坏消息是您遇到了一个非常棘手的问题.我已经在这个特定领域进行了大量研究，目前，您可以使用h.264解码器来真正使全帧大小变成2倍的帧缓冲区是唯一可行的方法.一旦将图像数据解码到实际的内存中，就可以使用OpenGL完成很多技巧(请看一下GPUImage).但是，最大的问题不是如何将像素从实时内存移动到屏幕上.真正的问题是如何将像素从磁盘上的编码形式移到活动内存中.人们可以使用文件映射的内存将像素保留在磁盘上，但是IO子系统的速度不够快，无法交换出足够的页面，因此无法从映射的内存中流式传输2倍全屏尺寸的图像.过去，此功能在全屏尺寸为1倍时效果很好，但现在2倍尺寸的屏幕实际上是内存量的4倍，硬件无法跟上.您也可以尝试将帧以更压缩的格式(例如PNG)存储在磁盘上.但是，然后解码压缩格式会将问题从IO绑定更改为CPU绑定，您仍然会遇到麻烦.请查看我的博客文章 opengl_write_texture_cache 以获得完整的源代码和计时结果我发现了这种方法.如果您有一个非常特定的格式，可以将输入图像数据限制为(例如8位表)，则可以使用GPU通过着色器将8位数据作为32BPP像素进行平移，如本示例xcode项目 opengl_color_cycle .但是，我的建议是看一下如何使用h.264解码器，因为它实际上可以在硬件中解码大量数据，并且没有其他方法可能会为您提供所需的结果.

So, the bad news is that you have run into a really hard problem. I have been doing quite a lot of research in this specific area and currently the only way that you can actually blit a framebuffer that is the size of the full screen at 2x is to use the h.264 decoder. There are quite a few nice tricks that can be done with OpenGL once you have image data already decoded into actual memory (take a look at GPUImage). But, the big problem is not how to move the pixels from live memory onto the screen. The real issue is how to move the pixels from the encoded form on disk into live memory. One can use file mapped memory to hold the pixels on disk, but the IO subsystem is not fast enough to be able to swap out enough pages to make it possible to stream 2x full screen size images from mapped memory. This used to work great with 1x full screen sizes, but now the 2x size screens are actually 4x the amount of memory and the hardware just cannot keep up. You could also try to store frames on disk in a more compressed format, like PNG. But, then decoding the compressed format changes the problem from IO bound to CPU bound and you are still stuck. Please have a look at my blog post opengl_write_texture_cache for the full source code and timing results I found with that approach. If you have a very specific format that you can limit the input image data to (like an 8 bit table), then you could use the GPU to blit 8 bit data as 32BPP pixels via a shader, as shown in this example xcode project opengl_color_cycle. But, my advice would be to look at how you could make use of the h.264 decoder since it is actually able to decode that much data in hardware and no other approaches are likely to give you the kind of results you are looking for.

这篇关于将RGB像素数据高性能复制到iOS中的屏幕的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！