在 iPhone GPU 上解码视频帧 [英] Decode video frames on iPhone GPU

问题描述

我正在寻找在 iPhone 上解码本地 mpeg-4 视频帧的最快方法.我只是对每 10 帧中像素的亮度值感兴趣.我不需要在任何地方渲染视频.

I'm looking for the fastest way to decode a local mpeg-4 video's frames on the iPhone. I'm simply interested in the luminance values of the pixels in every 10th frame. I don't need to render the video anywhere.

我尝试过 ffmpeg、AVAssetReader、ImageAssetGenerator、OpenCV 和 MPMoviePlayer，但它们都太慢了.我能得到的最快速度是 ~2x(一分钟扫描 2 分钟的视频).我想要接近 10 倍的东西.

I've tried ffmpeg, AVAssetReader, ImageAssetGenerator, OpenCV, and MPMoviePlayer but they're all too slow. The fastest speed I can get is ~2x (2 minutes of video scanned in a minute). I'd like something closer to 10x.

假设我上面的尝试没有使用 GPU，有什么方法可以通过在 GPU 上运行的东西来实现我的目标吗?OpenGL 似乎主要用于渲染输出，但我已经看到它用作传入视频的过滤器.也许这是一个选择?

Assuming my attempts above didn't utilize the GPU, is there any way to accomplish my goal with something that does run on the GPU? OpenGL seems like it's mostly for rendering output but I have seen it used as filters for incoming video. Maybe that's an option?

提前致谢！

推荐答案

如果您愿意使用仅限 iOS 5 的解决方案，请查看 2011 年 WWDC 会议上 AVCaputureSession 上的示例应用 ChromaKey.

If you are willing to use an iOS 5 only solution, take a look at the sample app ChromaKey from the 2011 WWDC session on AVCaputureSession.

该演示从内置摄像头捕获 30 FPS 的视频，并将每一帧作为纹理传递给 OpenGL.然后它使用 OpenGL 来操作帧，并可选择将结果写入输出视频文件.

That demo captures 30 FPS of video from the built-in camera and passes each frame to OpenGL as a texture. It then uses OpenGL to manipulate the frame, and optionally writes the result out to an output video file.

代码使用一些严重的低级魔法将核心视频像素缓冲区从 AVCaptureSession 绑定到 OpenGL，以便它们共享图形硬件中的内存.

The code uses some serious low-level magic to bind a Core Video Pixel buffer from an AVCaptureSession to OpenGL so they share memory in the graphics hardware.

将 AVCaptureSession 更改为使用电影文件作为输入而不是相机输入应该相当简单.

It should be fairly straightforward to change the AVCaptureSession to use a movie file as input rather than camera input.

您可以将会话设置为以 Y/UV 形式而不是 RGB 形式传递帧，其中 Y 分量是亮度.如果做不到这一点，编写一个将每个像素的 RGB 值转换为亮度值的着色器将是一件非常简单的事情.

You could probably set up the session to deliver frames in Y/UV form rather than RGB, where the Y component is luminance. Failing that, it would be a pretty simple matter to write a shader that would convert RGB values for each pixel to luminance values.

您应该能够在所有帧上执行所有这些操作，而不仅仅是每 10 帧.

You should be able to do all this on ALL Frames, not just every 10th frame.

这篇关于在 iPhone GPU 上解码视频帧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！