将ARFrame捕获的图像转换为UIImage方向问题 [英] Convert ARFrame's captured image to UIImage orientation issue

问题描述

我想检测球并让AR模型与其交互.我使用opencv进行球的检测并发送球的中心，可以在hitTest中使用它来获取sceneView中的坐标.我一直在使用以下功能将CVPixelBuffer转换为UIImage:

I want to detect ball and have AR model interact with it. I used opencv for ball detection and send center of ball which I can use in hitTest to get coordinates in sceneView. I have been converting CVPixelBuffer to UIImage using following function:

static func convertToUIImage(buffer: CVPixelBuffer) -> UIImage?{
    let ciImage = CIImage(cvPixelBuffer: buffer)
    let temporaryContext = CIContext(options: nil)
    if let temporaryImage = temporaryContext.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(buffer), height: CVPixelBufferGetHeight(buffer)))
    {
        let capturedImage = UIImage(cgImage: temporaryImage)
        return capturedImage
    }
    return nil
}

这给了我旋转的图像:

然后我发现了有关使用以下方法更改方向的信息:

Then i found about changing orientation using:

let capturedImage = UIImage(cgImage: temporaryImage, scale: 1.0, orientation: .right)

当设备处于纵向状态时，它给出了正确的方向，而将设备旋转至水平状态又给出了旋转后的图像.

While it gave correct orientation while device is in portrait, rotating device to landscape again gave rotated image.

现在我正在考虑使用viewWillTransition处理它.但在此之前，我想知道:

Now I am thinking about handling it using viewWillTransition. But before that i want to know:

1. 是否还有其他方法可以转换方向正确的图像?
2. 为什么会这样?

推荐答案

1.还有另一种方法来转换具有正确方向的图像吗?

您可以尝试使用ARSCNView的snapshot()(继承自SCNView)，

You may try to use snapshot() of ARSCNView (inherited from SCNView), which:

绘制视图的内容并将其作为新的图像对象返回

Draws the contents of the view and returns them as a new image object

因此，如果您有一个像这样的对象:

so if you have an object like:

@IBOutlet var arkitSceneView:ARSCNView!

您只需要这样做:

let imageFromArkitScene:UIImage? = arkitSceneView.snapshot()

---

2.为什么会发生这种情况?

这是因为CVPixelBuffer来自ARFrame，即:

通过正在运行的AR会话(连续)从设备相机捕获.

captured (continuously) from the device camera, by the running AR session.

好吧，由于摄像头的方向不会随设备的旋转而改变(它们是分开的)，为了能够将框的方向调整为当前视图，您应该重新调整从摄像头捕获的图像的方向应用用displayTransform(for:viewportSize:)提取的仿射变换:

Well, since the camera orientation does not change with the rotation of the device (they are separate), to be able to adjust the orientation of your frame to the current view, you should re-orient the image captured from your camera applying the affine transform extracted with displayTransform(for:viewportSize:):

返回仿射变换，用于在归一化的图像坐标和适合在屏幕上呈现相机图像的坐标空间之间进行转换.

Returns an affine transform for converting between normalized image coordinates and a coordinate space appropriate for rendering the camera image onscreen.

您可能会在此处，使用示例:

you may find good documentation here, usage example:

let orient = UIApplication.shared.statusBarOrientation
let viewportSize = yourSceneView.bounds.size
let transform = frame.displayTransform(for: orient, viewportSize: viewportSize).inverted()
var finalImage = CIImage(cvPixelBuffer: pixelBuffer).transformed(by: transform)

这篇关于将ARFrame捕获的图像转换为UIImage方向问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！