在MacOS中使用AVFoundation和Metal的帧速率非常慢 [英] Very slow framerate with AVFoundation and Metal in MacOS

问题描述

我正在尝试将Apple的 AVCamFilter 示例修改为MacOS.过滤似乎有效，但是通过Metal渲染处理后的图像使我获得了每帧几秒钟的帧速率.我尝试了不同的方法，但已经停留了很长时间.

这是项目 AVCamFilterMacOS -任何对AVFoundation with Metal有更好了解的人都可以告诉我怎么了?我一直在阅读文档，并练习如何显示未处理的图像，以及将其他模型(如模型)渲染到金属视图，但是我似乎无法使处理后的CMSampleBuffer以合理的帧速率渲染.

即使我跳过渲染器并将videoPixelBuffer直接发送到金属视图，该视图的性能也非常不稳定.

以下是我在控制器中使用的一些相关渲染代码:

  func captureOutput(_输出:AVCaptureOutput，didOutput sampleBuffer:CMSampleBuffer，来自连接:AVCaptureConnection){processVideo(sampleBuffer:sampleBuffer)} 

func processVideo(sampleBuffer:CMSampleBuffer){如果！renderingEnabled {返回}

  guard let videoPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)，让formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer)else {返回}如果！self.videoFilter.isPrepared {/*outputRetainedBufferCountHint是渲染器保留的像素缓冲区数.此值通知渲染器如何确定其缓冲池的大小以及要预分配多少个像素缓冲区.允许3帧延迟来覆盖dispatch_async调用.*/self.videoFilter.prepare(with:formatDescription，outputRetainedBufferCountHint:3)}//通过过滤器发送像素缓冲区警卫队让filteredBuffer = self.videoFilter.render(pixelBuffer:videoPixelBuffer)否则{打印(无法过滤视频缓冲区")返回}self.previewView.pixelBuffer =过滤缓冲区} 

从渲染器:

  func render(pixelBuffer:CVPixelBuffer)->CVPixelBuffer?{如果！isPrepared {assertionFailure(无效状态:未准备好.")返回零}var newPixelBuffer:CVPixelBuffer?CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault，outputPixelBufferPool !,& newPixelBuffer)保护卫队outputPixelBuffer = newPixelBuffer else {print(分配失败:无法从池中获取像素缓冲区.(\(self.description))")返回零}守护让inputTexture = makeTextureFromCVPixelBuffer(pixelBuffer:pixelBuffer，textureFormat:.bgra8Unorm)，让outputTexture = makeTextureFromCVPixelBuffer(pixelBuffer:outputPixelBuffer，textureFormat:.bgra8Unorm)else {返回零}//设置命令队列，缓冲区和编码器.警卫让commandQueue = commandQueue，让commandBuffer = commandQueue.makeCommandBuffer()，让commandEncoder = commandBuffer.makeComputeCommandEncoder()else {print(无法创建Metal命令队列.")CVMetalTextureCacheFlush(textureCache !, 0)返回零}commandEncoder.label =玫瑰金属"commandEncoder.setComputePipelineState(computePipelineState！)commandEncoder.setTexture(inputTexture，index:0)commandEncoder.setTexture(outputTexture，索引:1)//设置线程组.让width = computePipelineState！.threadExecutionWidth让height = computePipelineState！.maxTotalThreadsPerThreadgroup/width让threadsPerThreadgroup = MTLSizeMake(宽度，高度，1)让threadgroupsPerGrid = MTLSize(width:(inputTexture.width + width-1)/width，高度:(inputTexture.height + height-1)/高度，深度:1)commandEncoder.dispatchThreadgroups(threadgroupsPerGrid，threadsPerThreadgroup:threadsPerThreadgroup)commandEncoder.endEncoding()commandBuffer.commit()返回outputPixelBuffer}func makeTextureFromCVPixelBuffer(pixelBuffer:CVPixelBuffer，textureFormat:MTLPixelFormat)->MTLTexture?{让width = CVPixelBufferGetWidth(pixelBuffer)让高度= CVPixelBufferGetHeight(pixelBuffer)//从图像缓冲区创建金属纹理.var cvTextureOut:CVMetalTexture?CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault，textureCache，pixelBuffer，nil，textureFormat，宽度，高度，0和& cvTextureOut)警卫队让cvTexture = cvTextureOut，让纹理= CVMetalTextureGetTexture(cvTexture)其他{CVMetalTextureCacheFlush(textureCache，0)返回零}返回纹理} 

最后是金属视图:

 覆盖函数func draw(_ rect:CGRect){var pixelBuffer:CVPixelBuffer?var镜像=否var rotation:Rotation = .rotate0DegreessyncQueue.sync {pixelBuffer = internalPixelBuffer镜像=内部镜像旋转=内部旋转}守护让drawable = currentDrawable，让currentRenderPassDescriptor = currentRenderPassDescriptor，让PreviewPixelBuffer = pixelBuffer else {返回}//从图像缓冲区创建金属纹理.让width = CVPixelBufferGetWidth(previewPixelBuffer)让高度= CVPixelBufferGetHeight(previewPixelBuffer)如果textureCache == nil {createTextureCache()}var cvTextureOut:CVMetalTexture?CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault，textureCache !,PreviewPixelBuffer，零，.bgra8Unorm，宽度，高度，0，& cvTextureOut)警卫队让cvTexture = cvTextureOut，让纹理= CVMetalTextureGetTexture(cvTexture)其他{打印(无法创建预览纹理")CVMetalTextureCacheFlush(textureCache !, 0)返回}如果texture.width！= textureWidth ||texture.height！= textureHeight ||self.bounds！= internalBounds ||镜像！= textureMirroring ||旋转！= textureRotation {setupTransform(width:texture.width，height:texture.height，镜像:镜像，旋转:旋转)}//设置命令缓冲区和编码器警卫让commandQueue = commandQueue else {打印(无法创建金属命令队列")CVMetalTextureCacheFlush(textureCache !, 0)返回}警卫让commandBuffer = commandQueue.makeCommandBuffer()else {打印(无法创建金属命令缓冲区")CVMetalTextureCacheFlush(textureCache !, 0)返回}警卫让commandEncoder = commandBuffer.makeRenderCommandEncoder(descriptor:currentRenderPassDescriptor)else {打印(无法创建金属命令编码器")CVMetalTextureCacheFlush(textureCache !, 0)返回}commandEncoder.label =预览显示"commandEncoder.setRenderPipelineState(renderPipelineState！)commandEncoder.setVertexBuffer(vertexCoordBuffer，偏移量:0，索引:0)commandEncoder.setVertexBuffer(textCoordBuffer，偏移量:0，索引:1)commandEncoder.setFragmentTexture(纹理，索引:0)commandEncoder.setFragmentSamplerState(sampler，index:0)commandEncoder.drawPrimitives(类型:.triangleStrip，vertexStart:0，vertexCount:4)commandEncoder.endEncoding()//绘制到屏幕上.commandBuffer.present(drawable)commandBuffer.commit()} 

所有这些代码都在链接的项目中

解决方案

捕获设备委托不拥有在回调中接收的样本缓冲区，因此接收者有责任确保将其保留长达它们的内容是必需的.该项目目前无法确保.

相反，通过调用 CMSampleBufferGetImageBuffer 并将结果像素缓冲区包装在纹理中，视图控制器允许释放采样缓冲区，这意味着未定义对其相应像素缓冲区的未来操作./p>

确保样本缓冲区寿命足够长以进行处理的一种方法是向相机视图控制器类添加一个私有成员，该成员保留最近接收到的样本缓冲区:

  private var sampleBuffer:CMSampleBuffer！ 

，然后在调用 processVideo 之前，在 captureOutput(...)方法中设置此成员.您甚至不必进一步引用它.保留它的事实应该可以防止您看到的断断续续和难以预测的行为.

此解决方案可能并不完美，因为在捕获会话中断或其他暂停的情况下，其保留样本缓冲区的时间比严格保留的时间长.您可以设计自己的方案来管理对象的生命周期.重要的是要确保根样本缓冲区对象始终存在，直到您完成引用其内容的所有纹理为止.

I'm trying to adapt Apple's AVCamFilter sample to MacOS. The filtering appears to work, but rendering the processed image through Metal gives me a framerate of several seconds per frame. I've tried different approaches, but have been stuck for a long time.

This is the project AVCamFilterMacOS - Can anyone with better knowledge of AVFoundation with Metal tell me what's wrong? I've been reading the documentation and practicing getting the unprocessed image to display, as well as rendering other things like models to the metal view but I can't seem to get the processed CMSampleBuffer to render at a reasonable framerate.

Even if I skip the renderer and send the videoPixelBuffer to the metal view directly, the view's performance is pretty jittery.

Here is some of the relevant rendering code I'm using in the controller:

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    processVideo(sampleBuffer: sampleBuffer)
}

func processVideo(sampleBuffer: CMSampleBuffer) { if !renderingEnabled { return }

guard let videoPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer),
  let formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer) else {
    return
}

if !self.videoFilter.isPrepared {
  /*
   outputRetainedBufferCountHint is the number of pixel buffers the renderer retains. This value informs the renderer
   how to size its buffer pool and how many pixel buffers to preallocate. Allow 3 frames of latency to cover the dispatch_async call.
   */
  self.videoFilter.prepare(with: formatDescription, outputRetainedBufferCountHint: 3)
}

// Send the pixel buffer through the filter
guard let filteredBuffer = self.videoFilter.render(pixelBuffer: videoPixelBuffer) else {
  print("Unable to filter video buffer")
  return
}

self.previewView.pixelBuffer = filteredBuffer
}

And from the renderer:

func render(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
    if !isPrepared {
        assertionFailure("Invalid state: Not prepared.")
        return nil
    }

    var newPixelBuffer: CVPixelBuffer?
    CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, outputPixelBufferPool!, &newPixelBuffer)
    guard let outputPixelBuffer = newPixelBuffer else {
        print("Allocation failure: Could not get pixel buffer from pool. (\(self.description))")
        return nil
    }
    guard let inputTexture = makeTextureFromCVPixelBuffer(pixelBuffer: pixelBuffer, textureFormat: .bgra8Unorm),
        let outputTexture = makeTextureFromCVPixelBuffer(pixelBuffer: outputPixelBuffer, textureFormat: .bgra8Unorm) else {
            return nil
    }

    // Set up command queue, buffer, and encoder.
    guard let commandQueue = commandQueue,
        let commandBuffer = commandQueue.makeCommandBuffer(),
        let commandEncoder = commandBuffer.makeComputeCommandEncoder() else {
            print("Failed to create a Metal command queue.")
            CVMetalTextureCacheFlush(textureCache!, 0)
            return nil
    }

    commandEncoder.label = "Rosy Metal"
    commandEncoder.setComputePipelineState(computePipelineState!)
    commandEncoder.setTexture(inputTexture, index: 0)
    commandEncoder.setTexture(outputTexture, index: 1)

    // Set up the thread groups.
    let width = computePipelineState!.threadExecutionWidth
    let height = computePipelineState!.maxTotalThreadsPerThreadgroup / width
    let threadsPerThreadgroup = MTLSizeMake(width, height, 1)
    let threadgroupsPerGrid = MTLSize(width: (inputTexture.width + width - 1) / width,
                                      height: (inputTexture.height + height - 1) / height,
                                      depth: 1)
    commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup)

    commandEncoder.endEncoding()
    commandBuffer.commit()
    return outputPixelBuffer
}

func makeTextureFromCVPixelBuffer(pixelBuffer: CVPixelBuffer, textureFormat: MTLPixelFormat) -> MTLTexture? {
    let width = CVPixelBufferGetWidth(pixelBuffer)
    let height = CVPixelBufferGetHeight(pixelBuffer)

    // Create a Metal texture from the image buffer.
    var cvTextureOut: CVMetalTexture?
    CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, nil, textureFormat, width, height, 0, &cvTextureOut)

    guard let cvTexture = cvTextureOut, let texture = CVMetalTextureGetTexture(cvTexture) else {
        CVMetalTextureCacheFlush(textureCache, 0)

        return nil
    }

    return texture
}

And finally the metal view:

override func draw(_ rect: CGRect) {
    var pixelBuffer: CVPixelBuffer?
    var mirroring = false
    var rotation: Rotation = .rotate0Degrees

    syncQueue.sync {
        pixelBuffer = internalPixelBuffer
        mirroring = internalMirroring
        rotation = internalRotation
    }

    guard let drawable = currentDrawable,
        let currentRenderPassDescriptor = currentRenderPassDescriptor,
        let previewPixelBuffer = pixelBuffer else {
            return
    }

    // Create a Metal texture from the image buffer.
    let width = CVPixelBufferGetWidth(previewPixelBuffer)
    let height = CVPixelBufferGetHeight(previewPixelBuffer)

    if textureCache == nil {
        createTextureCache()
    }
    var cvTextureOut: CVMetalTexture?
    CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                              textureCache!,
                                              previewPixelBuffer,
                                              nil,
                                              .bgra8Unorm,
                                              width,
                                              height,
                                              0,
                                              &cvTextureOut)
    guard let cvTexture = cvTextureOut, let texture = CVMetalTextureGetTexture(cvTexture) else {
        print("Failed to create preview texture")

        CVMetalTextureCacheFlush(textureCache!, 0)
        return
    }

    if texture.width != textureWidth ||
        texture.height != textureHeight ||
        self.bounds != internalBounds ||
        mirroring != textureMirroring ||
        rotation != textureRotation {
        setupTransform(width: texture.width, height: texture.height, mirroring: mirroring, rotation: rotation)
    }

    // Set up command buffer and encoder
    guard let commandQueue = commandQueue else {
        print("Failed to create Metal command queue")
        CVMetalTextureCacheFlush(textureCache!, 0)
        return
    }

    guard let commandBuffer = commandQueue.makeCommandBuffer() else {
        print("Failed to create Metal command buffer")
        CVMetalTextureCacheFlush(textureCache!, 0)
        return
    }

    guard let commandEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: currentRenderPassDescriptor) else {
        print("Failed to create Metal command encoder")
        CVMetalTextureCacheFlush(textureCache!, 0)
        return
    }

    commandEncoder.label = "Preview display"
    commandEncoder.setRenderPipelineState(renderPipelineState!)
    commandEncoder.setVertexBuffer(vertexCoordBuffer, offset: 0, index: 0)
    commandEncoder.setVertexBuffer(textCoordBuffer, offset: 0, index: 1)
    commandEncoder.setFragmentTexture(texture, index: 0)
    commandEncoder.setFragmentSamplerState(sampler, index: 0)
    commandEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
    commandEncoder.endEncoding()

    // Draw to the screen.
    commandBuffer.present(drawable)
    commandBuffer.commit()
}

All of this code is in the linked project

解决方案

Capture device delegates don't own the sample buffers they receive in their callbacks, so it's incumbent on the receiver to make sure they're retained for as long as their contents are needed. This project doesn't currently ensure that.

Rather, by calling CMSampleBufferGetImageBuffer and wrapping the resulting pixel buffer in a texture, the view controller is allowing the sample buffer to be released, meaning that future operations on its corresponding pixel buffer are undefined.

One way to ensure the sample buffer lives long enough to be processed is to add a private member to the camera view controller class that retains the most-recently received sample buffer:

private var sampleBuffer: CMSampleBuffer!

and then set this member in the captureOutput(...) method before calling processVideo. You don't even have to reference it further; the fact that it's retained should prevent the stuttery and unpredictable behavior you're seeing.

This solution may not be perfect, since it retains the sample buffer for longer than strictly necessary in the event of a capture session interruption or other pause. You can devise your own scheme for managing object lifetimes; the important thing is to ensure that the root sample buffer object sticks around until you're done with any textures that refer to its contents.

这篇关于在MacOS中使用AVFoundation和Metal的帧速率非常慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！