在 iPhone X(SceneKit/ARKit 应用程序)上从 2D 区域获取面部标志点的深度很困难 [英] Difficulty getting depth of face landmark points from 2D regions on iPhone X (SceneKit/ARKit app)

问题描述

我正在 iPhone X 上使用前置摄像头运行面部标志检测，并且非常努力地获取面部标志的 3D 点(VNFaceLandmarkRegion2D 仅提供图像坐标 X、Y).

I'm running face landmark detection using the front-facing camera on iPhone X, and am trying very hard to get 3D points of face landmarks (VNFaceLandmarkRegion2D gives image coordinates X, Y only).

我一直在尝试使用 ARSCNView.hitTest 或 ARFrame.hitTest，但到目前为止都没有成功.我认为我的错误可能是将初始地标点转换为正确的坐标系.我已经尝试了很多排列，但目前根据我的研究，这是我想出的:

I've been trying to use either the ARSCNView.hitTest or ARFrame.hitTest, but am so far unsuccessful. I think my error may be in converting the initial landmark points to the correct coordinate system. I've tried quite a few permutations, but currently based on my research this is what I've come up with:

let point = CGPoint(x: landmarkPt.x * faceBounds.width + faceBounds.origin.x, y: (1.0 - landmarkPt.y) * faceBounds.height + faceBounds.origin.y)
let screenPoint = CGPoint(point.x * view.bounds.width, point.y * view.bounds.height)
frame.hitTest(screenPoint, types: ARHitTestResult.ResultType.featurePoint)

我也尝试过

let newPoint = CGPoint(x: point.x, y: 1.0 - point.y) 

转换后，但似乎没有任何效果.我的 frame.hitTest 结果总是空的.我在转换中遗漏了什么吗?

after the conversion, but nothing seems to work. My frame.hitTest result is always empty. Am I missing anything in the conversion?

前置摄像头是否为此增加了另一个层次?(我还尝试在某一点反转初始 X 值，以防坐标系被镜像).在我看来，初始地标 normalizedPoints 有时是负的，有时也大于 1.0，这对我来说没有任何意义.如果这很重要，我正在使用 ARSession.currentFrame?.capturedImage 来捕获前置摄像头的帧.

Does the front-facing camera add another level to this? (I also tried inverting the initial X value at one point, in case the coordinate system was being mirrored). It also seems to me that the initial landmark normalizedPoints are sometimes negative and also sometimes greater than 1.0, which doesn't make any sense to me. I'm using ARSession.currentFrame?.capturedImage to capture the frame of the front-facing camera, if that's important.

非常感谢任何帮助，非常感谢！

Any help would be very, very appreciated, thanks so much!

-- 已解决--

对于有类似问题的任何人:我终于得到了命中测试结果！

For anyone with similar issues: I am finally getting hit test results!

for point in observation.landmarks?.allPoints?.pointsInImage(imageSize: sceneView.bounds.size) {
    let result = ARSCNView.hitTest(point, options: [ARSCNHitTestOption.rootNode: faceNode)
}

我使用人脸几何作为附加到人脸节点的遮挡节点.

I use the face geometry as an occlusion node attached to the face node.

谢谢里克斯特！

推荐答案

您正在使用 ARFaceTrackingConfiguration，对吗?在这种情况下，featurePoint 命中测试类型将无济于事，因为特征点是世界跟踪的一部分，而不是面部跟踪……事实上，几乎所有 ARKit 命中测试机制特定于世界跟踪，对面部跟踪没有用.

You're using ARFaceTrackingConfiguration, correct? In that case, the featurePoint hit test type won't help you, because feature points are part of world tracking, not face tracking... in fact, just about all the ARKit hit testing machinery is specific to world tracking, and not useful to face tracking.

您可以做的是利用面部网格 (ARFaceGeometry) 和面部姿势跟踪 (ARFaceAnchor) 来从 2D 图像点到3D 世界空间(或相机空间)点.为此，您至少可以走几条路:

What you can do instead is make use of the face mesh (ARFaceGeometry) and face pose tracking (ARFaceAnchor) to work your way from a 2D image point to a 3D world-space (or camera-space) point. There's at least a couple paths you could go down for that:

1. 如果您已经在使用 SceneKit，则可以使用 SceneKit 的命中测试代替 ARKit.(也就是说，您正在针对在 SceneKit 中建模的虚拟"几何体进行命中测试，而不是针对由 ARKit 建模的真实世界环境的稀疏估计.在这种情况下，面部网格的虚拟"几何体通过ARKit.)也就是说，你想要 ARSCNView.hitTest(_:options:)(继承自 SCNSceneRenderer)，而不是 hitTest(_:types:).当然，这意味着您需要使用 ARSCNFaceGeometry 来可视化场景中的面部网格，以及 ARSCNView 的节点/锚点映射以使其跟踪面部姿势(尽管如果您希望视频图像显示出来，您可以使网格透明)——否则 SceneKit 命中测试将找不到任何 SceneKit 几何体.

1. If you're already using SceneKit, you can use SceneKit's hit testing instead of ARKit's. (That is, you're hit testing against "virtual" geometry modeled in SceneKit, not against a sparse estimate of the real-world environment modeled by ARKit. In this case, the "virtual" geometry of the face mesh comes into SceneKit via ARKit.) That is, you want ARSCNView.hitTest(_:options:) (inherited from SCNSceneRenderer), not hitTest(_:types:). Of course, this means you'll need to be using ARSCNFaceGeometry to visualize the face mesh in your scene, and ARSCNView's node/anchor mapping to make it track the face pose (though if you want the video image to show through, you can make the mesh transparent) — otherwise the SceneKit hit test won't have any SceneKit geometry to find.

如果您没有使用 SceneKit，或者由于某种原因无法将面部网格放入您的场景中，那么您将拥有针对面部网格重建命中测试所需的所有信息.ARCamera 有视图和投影矩阵，告诉您 2D 屏幕投影与 3D 世界空间的关系，ARFaceAnchor 告诉您人脸在世界空间中的位置，以及 ARFaceGeometry 告诉你每个点在脸上的位置——所以这只是一个一堆数学a> 从屏幕点到面网格点，反之亦然.

If you're not using SceneKit, or for some reason can't put the face mesh into your scene, you have all the information you need to reconstruct a hit test against the face mesh. ARCamera has view and projection matrices that tell you the relationship of your 2D screen projection to 3D world space, ARFaceAnchor tells you where the face is in world space, and ARFaceGeometry tells you where each point is on the face — so it's just a bunch of math to get from a screen point to a face-mesh point and vice versa.

这篇关于在 iPhone X(SceneKit/ARKit 应用程序)上从 2D 区域获取面部标志点的深度很困难的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！