MTKView以低于AVCaptureVideoPreviewLayer的分辨率显示摄像头提要 [英] MTKView displaying camera feed with lower resolution than AVCaptureVideoPreviewLayer

问题描述

我正在尝试将相机供稿流式传输到MTKView中，以将某些CI过滤器应用于实时流.初始化捕获会话并布局MTKView之后，以下是我设置金属(metalView是MTKview)的方法:

I'm trying to stream the camera feed into a MTKView for applying some CI filter to the live stream. After initializing the capture session and having layout the MTKView, here is how I set metal (metalView is the MTKview):

func setupMetal(){
    
    metalDevice = MTLCreateSystemDefaultDevice()
    metalView.device = metalDevice
    // Write when asked
    metalView.isPaused = true
    metalView.enableSetNeedsDisplay = false
    // Command queue for the GPU
    metalCommandQueue = metalDevice.makeCommandQueue()
    // Assign the delegate
    metalView.delegate = self
    // ???
    metalView.framebufferOnly = false
}

然后我从SampleBufferDelegate抓取帧并获得CIImage

I then grab frames from the SampleBufferDelegate and get a CIImage

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    //try and get a CVImageBuffer out of the sample buffer
    guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        return
    }
    
    //get a CIImage out of the CVImageBuffer
    let ciImage = CIImage(cvImageBuffer: cvBuffer)
    
    self.currentCIImage = ciImage
    
    // We draw to the metal view everytime we receive a frame
    metalView.draw()
            
}}

然后我使用currentCIImage通过其委托方法绘制MTKView:

I then use the currentCIImage to draw in the MTKView using its delegate methods:

extension ViewController : MTKViewDelegate {

func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    //tells us the drawable's size has changed
}

func draw(in view: MTKView) {
    //create command buffer for ciContext to use to encode it's rendering instructions to the GPU
    guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
        return
    }
    
    //make sure we actually have a ciImage to work with
    guard let ciImage = currentCIImage else {
        return
    }
    
    //make sure the current drawable object for this metal view is available (it's not in use by the previous draw cycle)
    guard let currentDrawable = view.currentDrawable else {
        return
    }
    
    //render into the metal texture
    // Check here if we find a more elegant solution for the bounds
    self.ciContext.render(ciImage,
                          to: currentDrawable.texture,
                          commandBuffer: commandBuffer,
                          bounds: CGRect(origin: .zero, size: view.drawableSize),
                          colorSpace: CGColorSpaceCreateDeviceRGB())
    
    //register where to draw the instructions in the command buffer once it executes
    commandBuffer.present(currentDrawable)
    //commit the command to the queue so it executes
    commandBuffer.commit()
}
}

它工作正常，我能够从MTKView中渲染的相机获取帧.但是，我注意到我没有获得完整的分辨率，因此无法在MTKview中缩放图像.我知道这与我如何设置捕获会话无关，因为当我使用标准AVCapturePreviewLayer时，一切都很好.关于我在做什么错的任何想法吗?

It works fine and I'm able to get frames from the camera rendered in the MTKView. However, I noticed that I'm not getting the full resolution, somehow the image is zoomed in the MTKview. I know it is not an issue related to how I set the capture session because when I use the standard AVCapturePreviewLayer it is all fine. Any ideas on what I'm doing wrong?

非常感谢！

PS该代码主要基于该出色的教程: https://betterprogramming.pub/using-cifilters-metal-to-make-a-custom-camera-in-ios-c76134993316 ，但对于我来说似乎不起作用

P.S This code is mainly based on this excellent tutorial: https://betterprogramming.pub/using-cifilters-metal-to-make-a-custom-camera-in-ios-c76134993316 but somehow it doesn't seem to work for me.

推荐答案

根据捕获会话的设置，摄像机帧的大小将与您的 MTKView 的大小不同.这意味着您需要缩放和转换它们，然后再渲染以匹配 currentDrawable 的大小.为此，我使用以下代码(在 render 调用之前的 draw 内部):

Depending on the setup of the capture session, the camera frames will not have the same size as your MTKView. That means you need to scale and translate them before rendering to match the size of the currentDrawable. I use the following code for that (inside draw, just before the render call):

    // scale to fit into view
    let drawableSize = self.drawableSize
    let scaleX = drawableSize.width / input.extent.width
    let scaleY = drawableSize.height / input.extent.height
    let scale = min(scaleX, scaleY)
    let scaledImage = input.transformed(by: CGAffineTransform(scaleX: scale, y: scale))

    // center in the view
    let originX = max(drawableSize.width - scaledImage.extent.size.width, 0) / 2
    let originY = max(drawableSize.height - scaledImage.extent.size.height, 0) / 2
    let centeredImage = scaledImage.transformed(by: CGAffineTransform(translationX: originX, y: originY))

这篇关于MTKView以低于AVCaptureVideoPreviewLayer的分辨率显示摄像头提要的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！