如何在CVImageBuffer中叠加每个捕获帧的视图，实时而不是后期处理 [英] How to superimpose views over each captured frame inside CVImageBuffer, realtime not post process

问题描述

我设法设置了一个基本的 AVCaptureSession ，它使用 AVCaptureFileOutputRecordingDelegate 记录视频并将其保存在设备上。我一直在搜索文档，以了解我们如何在正在录制的视频之上添加统计信息叠加。

I have managed to setup a basic AVCaptureSession which records a video and saves it on device by using AVCaptureFileOutputRecordingDelegate. I have been searching through docs to understand how we can add statistics overlays on top of the video which is being recorded.

ie

如上图所示。我在视频预览图层上有多个叠加层。现在，当我保存视频输出时，我也希望将这些视图组合到视频上。

As you can see in the above image. I have multiple overlays on top of video preview layer. Now, when I save my video output I would like to compose those views onto the video as well.

到目前为止我尝试了什么？

What have I tried so far?

• 老实说，我只是在互联网上跳来寻找一个有信誉的博客，解释如何做到这一点。但是找不到一个。


• 我已经阅读了一些可以渲染文本图层叠加的地方，如关注帖子，创建 CALayer 并将其添加为子图层。


• 但是，如果我想在正在录制的视频上呈现 MapView 呢？另外，我不是在寻找屏幕截图。屏幕上的某些内容不会成为最终录制内容的一部分，因此我希望能够选择将要编写的视图。

• Honestly, I am just jumping around on internet to find a reputable blog explaining how one would do this. But failed to find one.
• I have read few places that one could render text layer overlays as described in following post by creating CALayer and adding it as a sublayer.
• But, what about if I want to render MapView on top of the video being recorded. Also, I am not looking for screen capture. Some of the content on the screen will not be part of the final recording so I want to be able to cherry pick view that will be composed.

我在寻找什么？

1. 方向。


2. 没有直接解决方案


3. 文档链接和类名我应该阅读更多关于创建它的信息。

进展到目前为止：

我已经理解我需要得到从 CMSampleBuffer 中持有 CVImageBuffer 并在其上绘制文字。有些事情我还不清楚是否有可能以某种方式将MapView覆盖在正在录制的视频上。

I have managed to understand that I need to get hold of CVImageBuffer from CMSampleBuffer and draw text over it. There are things still unclear to me whether it is possible to somehow overlay MapView over the video that is being recorded.

推荐答案

最好的帮助您实现目标的方法是使用 Metal 框架。使用 Metal 相机有助于最大限度地减少对设备有限计算资源的影响。如果您尝试实现对摄像头传感器的最低开销访问，使用 AVCaptureSession 将是一个非常好的开始。

The best way that helps you to achieve your goal is to use a Metal framework. Using a Metal camera is good for minimising the impact on device’s limited computational resources. If you are trying to achieve the lowest-overhead access to camera sensor, using a AVCaptureSession would be a really good start.

您需要从 CMSampleBuffer 中获取每个帧数据（您是对的），然后将帧转换为a MTLTexture 。 AVCaptureSession 将通过委托回调从设备的相机连续发送帧。

You need to grab each frame data from CMSampleBuffer (you're right) and then to convert a frame to a MTLTexture. AVCaptureSession will continuously send us frames from device’s camera via a delegate callback.

所有可用的叠加层必须转换为 MTLTextures 也是。然后，您可以使用对操作合并所有 MTLTextures 图层。

All available overlays must be converted to MTLTextures too. Then you can composite all MTLTextures layers with over operation.

所以，在这里你可以找到所有必要信息四部分 金属相机 系列。

So, here you'll find all necessary info in four-part Metal Camera series.

此处是指向博客的链接 ：关于金属合成。

另外，我想发布代码的摘录（在Metal中使用AVCaptureSession）：

Also, I'd like to publish code's excerpt (working with AVCaptureSession in Metal):

import Metal

guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    // Handle an error here.
}

// Texture cache for converting frame images to textures
var textureCache: CVMetalTextureCache?

// `MTLDevice` for initializing texture cache
var metalDevice = MTLCreateSystemDefaultDevice()

guard
    let metalDevice = metalDevice
    where CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice, nil, &textureCache) == kCVReturnSuccess
else {
    // Handle an error (failed to create texture cache)
}

let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)

var imageTexture: CVMetalTexture?
let result = CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache.takeUnretainedValue(), imageBuffer, nil, pixelFormat, width, height, planeIndex, &imageTexture)

// `MTLTexture` is in the `texture` variable now.
guard
    let unwrappedImageTexture = imageTexture,
    let texture = CVMetalTextureGetTexture(unwrappedImageTexture),
    result == kCVReturnSuccess
else {
    throw MetalCameraSessionError.failedToCreateTextureFromImage
}

在这里你可以找到GitHub上的最终项目： MetalRenderCamera

这篇关于如何在CVImageBuffer中叠加每个捕获帧的视图，实时而不是后期处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！