如何将视觉框架坐标系转化为ARKit? [英] How to transform vision framework coordinate system into ARKit?

问题描述

我正在使用 ARKit(带有 SceneKit)来添加虚拟对象(例如球).我通过使用 Vision 框架并在视觉请求完成处理程序方法中接收其更新的位置来跟踪真实世界对象(例如脚).

I am using ARKit (with SceneKit) to add the virtual object (e.g. ball). I am tracking real world object (e.g. foot) by using Vision framework and receiving its updated position in vision request completion handler method.

let request = VNTrackObjectRequest(detectedObjectObservation: lastObservation, completionHandler: self.handleVisionRequestUpdate)

我想用虚拟替换跟踪的真实世界对象(例如用立方体替换脚)，但我不确定如何将 boundingBox rect(我们在视觉请求完成中收到)替换为场景套件节点，因为坐标系是不一样.

I wants to replace the tracked real world object with virtual (for example replace foot with cube) but I am not sure how to replace the boundingBox rect (which we receive in vision request completion) into scene kit node as coordinate system are different.

以下是视觉请求完成处理程序的代码:

below is the code of vision request completion handler:

 private func handleVisionRequestUpdate(_ request: VNRequest, error: Error?) {
    // Dispatch to the main queue because we are touching non-atomic, non-thread safe properties of the view controller
    DispatchQueue.main.async {
      // make sure we have an actual result
      guard let newObservation = request.results?.first as? VNDetectedObjectObservation else { return }

      // prepare for next loop
      self.lastObservation = newObservation

      // check the confidence level before updating the UI
      guard newObservation.confidence >= 0.3 else {
        return
      }

      // calculate view rect
      var transformedRect = newObservation.boundingBox

     //How to convert transformedRect into AR Coordinate
  self.node.position = SCNVector3Make(?.worldTransform.columns.3.x,
    ?.worldTransform.columns.3.y,

    }
  }

请指导我传递坐标系.

推荐答案

假设矩形在水平面上，您可以对场景的所有 4 个角进行命中测试，并使用其中 3 个角来计算宽度，矩形的高度、中心和方向.

Assuming the rectangle is on a horizontal plane, you can perform a hit test against the scene on all 4 corners and use 3 of those corners to calculate the width, height, center, and orientation of the rectangle.

我在 GitHub 上有一个演示应用程序可以做到这一点:https://github.com/mludowise/ARKitRectangleDetection

I have a demo app available on GitHub that does exactly that: https://github.com/mludowise/ARKitRectangleDetection

VNRectangleObservation 中矩形角的坐标将与图像的大小相关，并根据手机的旋转在不同的坐标中.您需要将它们乘以视图大小并根据手机的旋转反转它们:

The coordinates for the rectangle corners from VNRectangleObservation will be relative to the size of the image and in different coordinates depending on the phone's rotation. You'll need multiply them by the view size and invert them based on the phone's rotation:

func convertFromCamera(_ point: CGPoint, view sceneView: ARSCNView) -> CGPoint {
    let orientation = UIApplication.shared.statusBarOrientation

    switch orientation {
    case .portrait, .unknown:
        return CGPoint(x: point.y * sceneView.frame.width, y: point.x * sceneView.frame.height)
    case .landscapeLeft:
        return CGPoint(x: (1 - point.x) * sceneView.frame.width, y: point.y * sceneView.frame.height)
    case .landscapeRight:
        return CGPoint(x: point.x * sceneView.frame.width, y: (1 - point.y) * sceneView.frame.height)
    case .portraitUpsideDown:
        return CGPoint(x: (1 - point.y) * sceneView.frame.width, y: (1 - point.x) * sceneView.frame.height)
    }
}

然后您可以对所有 4 个角执行命中测试.在执行命中测试时使用 .existingPlaneUsingExtent 类型很重要，以便 ARKit 返回水平面的命中.

Then you can perform a hit test on all 4 corners. It's important to use the type .existingPlaneUsingExtent when performing the hit test so that ARKit returns hits for horizontal planes.

let tl = sceneView.hitTest(convertFromCamera(rectangle.topLeft, view: sceneView), types: .existingPlaneUsingExtent)
let tr = sceneView.hitTest(convertFromCamera(rectangle.topRight, view: sceneView), types: .existingPlaneUsingExtent)
let bl = sceneView.hitTest(convertFromCamera(rectangle.bottomLeft, view: sceneView), types: .existingPlaneUsingExtent)
let br = sceneView.hitTest(convertFromCamera(rectangle.bottomRight, view: sceneView), types: .existingPlaneUsingExtent)

然后它变得有点复杂......

Then it gets a little complicated...

因为每个命中测试可能返回 0 到 n 个结果，所以您需要过滤掉包含在不同平面上的任何命中测试.您可以通过比较每个 ARHitTestResult 的锚点来做到这一点:

Because each hit test could return with 0 to n results, you will need to filter out any hit tests that are contained on a different plane. You can do this by comparing the anchors for each ARHitTestResult:

hit1.anchor == hit2.anchor

此外，您只需要 4 个角中的 3 个即可识别矩形的尺寸、位置和方向，因此如果一个角不返回任何命中测试结果也没关系.看看这里我是如何做到的.

Also, you only need 3 out of 4 corners to identify the rectangle's dimensions, position, and orientation so it's okay if one corner doesn't return any hit test results. Take a look here for how I did that.

您可以根据左右角(顶部或底部)之间的距离计算矩形的宽度.同样，您可以从顶部和顶部之间的距离计算矩形的高度.底角(左或右).

You can calculate the rectangle's width from the distance between the left and right corners (for either top or bottom). Likewise you can calculate the rectangle's height from the distance between the top & bottom corners (for either left or right).

func distance(_ a: SCNVector3, from b: SCNVector3) -> CGFloat {
    let deltaX = a.x - b.x
    let deltaY = a.y - b.y
    let deltaZ = a.z - b.z

    return CGFloat(sqrt(deltaX * deltaX + deltaY * deltaY + deltaZ * deltaZ))
}

let width = distance(right, from: left)
let height = distance(top, from: bottom)

您可以通过获取矩形对角的中点(topLeft &bottomRight 或 topRight &bottomLeft)来计算其位置:

You can calculate its position by getting the midpoint from the opposite corners of the rectangle (either topLeft & bottomRight or topRight & bottomLeft):

let midX = (c1.x + c2.x) / 2
let midY = (c1.y + c2.y) / 2
let midZ = (c1.z + c2.z) / 2
let center = SCNVector3Make(midX, midY, midZ)

您还可以从左右角(顶部或底部)计算矩形的方向(沿 y 轴旋转):

You can also calculate the orientation of the rectangle (rotation along the y-axis) from the left and right corners (for either top or bottom):

let distX = right.x - left.x
let distZ = right.z - left.z
let orientation = -atan(distZ / distX)

然后将所有这些放在一起，并在 AR 中显示一些叠加在矩形上的内容.这是通过子类化 SCNNode 来显示虚拟矩形的示例:

Then put that all together and display something in AR overlaid on the rectangle. Here's an example of displaying a virtual rectangle by subclassing SCNNode:

class RectangleNode: SCNNode {

    init(center: SCNVector3, width: CGFloat, height: CGFloat, orientation: Float) {
        super.init()

        // Create the 3D plane geometry with the dimensions calculated from corners
        let planeGeometry = SCNPlane(width: width, height: height)
        let rectNode = SCNNode(geometry: planeGeometry)

        // Planes in SceneKit are vertical by default so we need to rotate
        // 90 degrees to match planes in ARKit
        var transform = SCNMatrix4MakeRotation(-Float.pi / 2.0, 1.0, 0.0, 0.0)

        // Set rotation to the corner of the rectangle
        transform = SCNMatrix4Rotate(transform, orientation, 0, 1, 0)

        rectNode.transform = transform

        // We add the new node to ourself since we inherited from SCNNode
        self.addChildNode(rectNode)

        // Set position to the center of rectangle
        self.position = center
    }
}

这篇关于如何将视觉框架坐标系转化为ARKit?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！