iOS恢复相机投影 [英] iOS revert camera projection
问题描述
我正在尝试估计与QR码在太空中相关的设备位置.我正在使用iOS11中都引入的ARKit和Vision框架，但是这个问题的答案可能并不取决于它们.
I'm trying to estimate my device position related to a QR code in space. I'm using ARKit and the Vision framework, both introduced in iOS11, but the answer to this question probably doesn't depend on them.
借助Vision框架，我能够获得在相机框架中界定QR码的矩形.我想将此矩形与从标准位置转换QR码所需的设备平移和旋转进行匹配.
With the Vision framework, I'm able to get the rectangle that bounds a QR code in the camera frame. I'd like to match this rectangle to the device translation and rotation necessary to transform the QR code from a standard position.
例如，如果我观察到框架:
For instance if I observe the frame:
* *
B
C
A
D
* *
如果我距离QR码1m，以它为中心，并假设QR码的边长为10cm，我会看到:
while if I was 1m away from the QR code, centered on it, and assuming the QR code has a side of 10cm I'd see:
* *
A0 B0
D0 C0
* *
这两个帧之间的设备转换是什么?我知道，可能无法获得确切的结果，因为观察到的QR码可能略微不是平面的，我们正在尝试估算并非完美的事物的仿射变换.
what has been my device transformation between those two frames? I understand that an exact result might not be possible, because maybe the observed QR code is slightly non planar and we're trying to estimate an affine transform on something that is not one perfectly.
我想sceneView.pointOfView?.camera?.projectionTransform比sceneView.pointOfView?.camera?.projectionTransform?.camera.projectionMatrix更有帮助，因为后者已经考虑了从ARKit推断出的变换，我对此问题不感兴趣.
I guess the sceneView.pointOfView?.camera?.projectionTransform is more helpful than the sceneView.pointOfView?.camera?.projectionTransform?.camera.projectionMatrix since the later already takes into account transform inferred from the ARKit that I'm not interested into for this problem.
我要如何填充
func get transform(
qrCodeRectangle: VNBarcodeObservation,
cameraTransform: SCNMatrix4) {
// qrCodeRectangle.topLeft etc is the position in [0, 1] * [0, 1] of A0
// expected real world position of the QR code in a referential coordinate system
let a0 = SCNVector3(x: -0.05, y: 0.05, z: 1)
let b0 = SCNVector3(x: 0.05, y: 0.05, z: 1)
let c0 = SCNVector3(x: 0.05, y: -0.05, z: 1)
let d0 = SCNVector3(x: -0.05, y: -0.05, z: 1)
let A0, B0, C0, D0 = ?? // CGPoints representing position in
// camera frame for camera in 0, 0, 0 facing Z+
// then get transform from 0, 0, 0 to current position/rotation that sees
// a0, b0, c0, d0 through the camera as qrCodeRectangle
}
====编辑====
====Edit====
尝试了很多事情之后，我最终使用openCV投影和透视求解器进行相机姿态估计，solvePnP这给了我一个旋转和平移，该旋转和平移应该代表QR码参考中的相机姿态.但是，当使用这些值并放置与逆变换相对应的对象时，QR码应位于相机空间中，我得到的偏移值不准确，并且无法使旋转工作:
After trying number of things, I ended up going for camera pose estimation using openCV projection and perspective solver, solvePnP This gives me a rotation and translation that should represent the camera pose in the QR code referential. However when using those values and placing objects corresponding to the inverse transformation, where the QR code should be in the camera space, I get inaccurate shifted values, and I'm not able to get the rotation to work:
// some flavor of pseudo code below
func renderer(_ sender: SCNSceneRenderer, updateAtTime time: TimeInterval) {
guard let currentFrame = sceneView.session.currentFrame, let pov = sceneView.pointOfView else { return }
let intrisics = currentFrame.camera.intrinsics
let QRCornerCoordinatesInQRRef = [(-0.05, -0.05, 0), (0.05, -0.05, 0), (-0.05, 0.05, 0), (0.05, 0.05, 0)]
// uses VNDetectBarcodesRequest to find a QR code and returns a bounding rectangle
guard let qr = findQRCode(in: currentFrame) else { return }
let imageSize = CGSize(
width: CVPixelBufferGetWidth(currentFrame.capturedImage),
height: CVPixelBufferGetHeight(currentFrame.capturedImage)
)
let observations = [
qr.bottomLeft,
qr.bottomRight,
qr.topLeft,
qr.topRight,
].map({ (imageSize.height * (1 - $0.y), imageSize.width * $0.x) })
// image and SceneKit coordinated are not the same
// replacing this by:
// (imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
// weirdly fixes an issue, see below
let rotation, translation = openCV.solvePnP(QRCornerCoordinatesInQRRef, observations, intrisics)
// calls openCV solvePnP and get the results
let positionInCameraRef = -rotation.inverted * translation
let node = SCNNode(geometry: someGeometry)
pov.addChildNode(node)
node.position = translation
node.orientation = rotation.asQuaternion
}
以下是输出:
其中A，B，C，D是按传递到程序的顺序的QR码角.
where A, B, C, D are the QR code corners in the order they are passed to the program.
当手机旋转时，预测的原点会保留在原位，但会从应有的位置移开.出乎意料的是，如果我改变观测值，我就能纠正这个问题:
The predicted origin stays in place when the phone rotates, but it's shifted from where it should be. Surprisingly, if I shift the observations values, I'm able to correct this:
// (imageSize.height * (1 - $0.y), imageSize.width * $0.x)
// replaced by:
(imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
，现在预测的起点稳固地保持在原位.但是我不知道移位值从哪里来.
and now the predicted origin stays robustly in place. However I don't understand where the shift values come from.
最后，我试图相对于QR码参考定位固定的方向:
Finally, I've tried to get an orientation fixed relatively to the QR code referential:
var n = SCNNode(geometry: redGeometry)
node.addChildNode(n)
n.position = SCNVector3(0.1, 0, 0)
n = SCNNode(geometry: blueGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0.1, 0)
n = SCNNode(geometry: greenGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0, 0.1)
当我直视QR码时方向很好，但是随后它的移动似乎与电话旋转有关:
The orientation is fine when I look at the QR code straight, but then it shifts by something that seems to be related to the phone rotation:
我有未解决的问题:
- 我该如何解决轮换问题?
- 位置偏移值从哪里来?
- 旋转，平移，QRCornerCoordinatesInQRRef，观测值，本征函数可以验证哪些简单关系?是O〜K ^ -1 *(R_3x2 | T)Q吗?因为如果是这样的话，那会偏离几个数量级.
如果有帮助，请参考以下几个数值:
If that's helpful, here are a few numerical values:
Intrisics matrix
Mat 3x3
1090.318, 0.000, 618.661
0.000, 1090.318, 359.616
0.000, 0.000, 1.000
imageSize
1280.0, 720.0
screenSize
414.0, 736.0
==== Edit2 ====
==== Edit2 ====
我注意到，当手机水平平行于QR码时，旋转效果很好(即旋转矩阵为[[a，0，b]，[0，1，0]，[c，0， d]])，无论实际的QR码方向是什么:
I've noticed that the rotation works fine when the phone stays horizontally parallel to the QR code (ie the rotation matrix is [[a, 0, b], [0, 1, 0], [c, 0, d]]), no matter what the actual QR code orientation is:
其他轮换无效.
推荐答案
我认为问题不在矩阵中.在顶点位置.要跟踪2D图像，您需要逆时针放置ABCD顶点(起点是位于虚构原点 x:0, y:0中的A顶点).我认为 VNRectangleObservation 类上的Apple文档(有关通过图像分析检测到的投影矩形区域的信息要求)含糊不清.您按照与官方文档中相同的顺序放置了顶点:
I suppose the problem is not in matrix. It's in vertices placement. For tracking 2D images you need to place ABCD vertices counter-clockwise (the starting point is A vertex located in imaginary origin x:0, y:0). I think Apple Documentation on VNRectangleObservation class (info about projected rectangular regions detected by an image analysis request) is vague. You placed your vertices in the same order as is in official documentation:
var bottomLeft: CGPoint
var bottomRight: CGPoint
var topLeft: CGPoint
var topRight: CGPoint
但是它们的放置方式必须与笛卡尔坐标系中出现的正旋转方向(绕Z轴)相同.
But they need to be placed the same way like positive rotation direction (about Z axis) occurs in Cartesian coordinates system:
ARKit(以及SceneKit和Vision中)的世界坐标空间始终遵循right-handed convention(正Y轴指向上方，正Z轴指向观察者，正X轴指向查看者的右侧)，但基于会话的配置进行定向.摄影机可在本地坐标空间中工作.
World Coordinate Space in ARKit (as well as in SceneKit and Vision) always follows a right-handed convention (the positive Y axis points upward, the positive Z axis points toward the viewer and the positive X axis points toward the viewer's right), but is oriented based on your session's configuration. Camera works in Local Coordinate Space.
围绕任何轴的旋转方向为正(逆时针)和负(顺时针).对于在ARKit和Vision中进行跟踪，这一点至关重要.
Rotation direction about any axis is positive (Counter-Clockwise) and negative (Clockwise). For tracking in ARKit and Vision it's critically important.
旋转顺序也很有意义. ARKit和SceneKit均以相反的顺序相对于节点的数据透视属性应用旋转:首先roll(关于Z轴)，然后是yaw(关于Y轴)，然后是X轴).因此轮换顺序为ZYX.
The order of rotation also makes sense. ARKit, as well as SceneKit, applies rotation relative to the node’s pivot property in the reverse order of the components: first roll (about Z axis), then yaw (about Y axis), then pitch (about X axis). So the rotation order is ZYX.
此外，还有关于 Nukepedia上的矩阵运算.
这篇关于iOS恢复相机投影的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
