将 ARFrame 的捕获图像转换为 UIImage 方向问题 [英] Convert ARFrame's captured image to UIImage orientation issue

问题描述

我想检测球并让 AR 模型与之交互.我使用 opencv 进行球检测并发送球中心，我可以在 hitTest 中使用它来获取 sceneView 中的坐标.我一直在使用以下函数将 CVPixelBuffer 转换为 UIImage:

I want to detect ball and have AR model interact with it. I used opencv for ball detection and send center of ball which I can use in hitTest to get coordinates in sceneView. I have been converting CVPixelBuffer to UIImage using following function:

static func convertToUIImage(buffer: CVPixelBuffer) -> UIImage?{
    let ciImage = CIImage(cvPixelBuffer: buffer)
    let temporaryContext = CIContext(options: nil)
    if let temporaryImage = temporaryContext.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(buffer), height: CVPixelBufferGetHeight(buffer)))
    {
        let capturedImage = UIImage(cgImage: temporaryImage)
        return capturedImage
    }
    return nil
}

这给了我旋转的图像:

然后我发现使用以下方法更改方向:

Then i found about changing orientation using:

let capturedImage = UIImage(cgImage: temporaryImage, scale: 1.0, orientation: .right)

虽然它在设备处于纵向时给出了正确的方向，但再次将设备旋转到横向会给出旋转的图像.

While it gave correct orientation while device is in portrait, rotating device to landscape again gave rotated image.

现在我正在考虑使用 viewWillTransition 处理它.但在此之前我想知道:

Now I am thinking about handling it using viewWillTransition. But before that i want to know:

1. 如果有其他方法可以转换具有正确方向的图像?
2. 为什么会发生这种情况?

推荐答案

1.有没有其他方法可以将图像转换为正确的方向?

你可以尝试使用ARSCNView的snapshot()(继承自SCNView)，它:

You may try to use snapshot() of ARSCNView (inherited from SCNView), which:

绘制视图的内容并将它们作为新的图像对象返回

Draws the contents of the view and returns them as a new image object

所以如果你有一个像:

@IBOutlet var arkitSceneView:ARSCNView!

你只需要这样做:

let imageFromArkitScene:UIImage? = arkitSceneView.snapshot()

<小时>

2.为什么会发生这种情况?

这是因为CVPixelBuffer来自ARFrame，即:

由正在运行的 AR 会话从设备摄像头(连续)捕获.

captured (continuously) from the device camera, by the running AR session.

好吧，由于相机方向不会随着设备的旋转而改变(它们是分开的)，为了能够将框架的方向调整为当前视图，您应该重新定向从相机捕获的图像应用使用 displayTransform(for:viewportSize:):

Well, since the camera orientation does not change with the rotation of the device (they are separate), to be able to adjust the orientation of your frame to the current view, you should re-orient the image captured from your camera applying the affine transform extracted with displayTransform(for:viewportSize:):

返回一个仿射变换，用于在标准化图像坐标和适合在屏幕上渲染相机图像的坐标空间之间进行转换.

Returns an affine transform for converting between normalized image coordinates and a coordinate space appropriate for rendering the camera image onscreen.

你可以在这里找到很好的文档，使用示例:>

you may find good documentation here, usage example:

let orient = UIApplication.shared.statusBarOrientation
let viewportSize = yourSceneView.bounds.size
let transform = frame.displayTransform(for: orient, viewportSize: viewportSize).inverted()
var finalImage = CIImage(cvPixelBuffer: pixelBuffer).transformed(by: transform)

这篇关于将 ARFrame 的捕获图像转换为 UIImage 方向问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！