用于 OpenGL 中头部跟踪的斜视锥体/离轴投影 [英] Skewed frustum/off-axis projection for head tracking in OpenGL

问题描述

我正在尝试在我的应用程序中进行离轴投影，并尝试根据用户的头部位置改变场景的视角.通常，鉴于我必须在屏幕上绘制一个框，我会在屏幕上绘制一个框:

I am trying to do an off-axis projection in my application and trying to change the perspective of the scene as per the user's head position. Normally, given that I had to draw a box on the screen, I would draw a Box on the screen as:

ofBox(350,250,0,50); //ofBox(x, y, z, size); where x, y and z used here are the screen coordinates

要在这里进行离轴投影，我知道我必须按如下方式更改透视投影:

To do an off-axis projection here, I am aware that I would have to change the perspective projection as follows:

vertFov = 0.5; near = 0.5; aspRatio = 1.33;
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glFrustum(near * (-vertFov * aspRatio + headX),
          near * (vertFov * aspRatio + headX),
          near * (-vertFov + headY),
          near * (vertFov + headY),
          near, far); //frustum changes as per the position of headX and headY
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
gluLookAt(headX * headZ, headY * headZ, 0, headX * headZ, headY * headZ, -1);
glTranslate(0,0,headZ);

对于上述情况下的对称截头体(headX 和 headY 为零)，left, right 参数为 -0.33, 0.33 和 bottom, top 参数出来是 -0.25, 0.25 并沿着这些坐标建立我的裁剪体积.我尝试使用鼠标模拟离轴进行测试并执行以下操作:

For a symmetric frustum in the above case (where headX and headY is zero), the left, right params come out to be -0.33, 0.33 and bottom, top parameters come out to be -0.25, 0.25 and establish my clipping volume along those coordinates. I tried to simulate the off-axis using a mouse for a test and did the following:

double mouseXPosition = (double)ofGetMouseX();
double mouseYPosition = (double)ofGetMouseY();
double scrWidth = (double)ofGetWidth();
double scrHeight = (double)ofGetHeight();

headX = ((scrWidth -mouseXPosition) / scrWidth) - 0.5;
headY = (mouseYPosition / scrHeight) - 0.5;
headZ = -0.5; //taken z constant for this mouse test

但是，我打算使用 Kinect 它为我提供了 (200, 400, 1000), (-250, 600, 1400), (400, 100, 1400) 等，当我有这些头部位置时，我无法弄清楚如何更改视锥体参数.例如:考虑到 0 位于 Kinect 的中心，如果用户移动使得他的位置是 (200, 400, 1000)，那么平截头体将如何这里参数有变化吗?
当还必须考虑从 Kinect 获得的 z-distance 时，如何绘制对象?随着 z 的增加，对象的尺寸必须变小，这可能通过上述离轴代码中的 glTrasnlate() 调用发生，但坐标系的两个比例是不同(glFrustum 现在将剪辑音量设置为 [-0.25,0.33] 到 [0.25,-0.33]，其中 Kinect 的数量级为数百 (400,200,1000)).那么如何将 z 值应用于 glFrustum/gluLookAt 呢?

However, I intend to use Kinect which gives me coordinates for head of the order of (200, 400, 1000), (-250, 600, 1400), (400, 100, 1400) etc. and I am not able to make out how to change the frustum parameters when I have those head positions. For eg: Considering 0 to be at the center for the Kinect, if the user moves such that his position is (200, 400, 1000), then how would the frustum parameters change here?
How will the objects have to be drawn when the z-distance obtained from Kinect will also have to be taken into account? Objects have to become smaller in size as z increase and that could happen by glTrasnlate() call inside the above off-axis code, but the two scales of the coordinate systems are different (glFrustum now sets clipping volume to [-0.25,0.33] to [0.25,-0.33] wheres Kinect is in the order of hundreds (400,200,1000)). How do I apply the z values to glFrustum/gluLookAt then?

推荐答案

首先，您不想使用 gluLookAt.gluLookAt 旋转相机，但用户看到的物理屏幕不会旋转.gluLookAt 仅在屏幕旋转以使屏幕法线继续指向用户时才起作用.离轴投影的透视失真将处理我们需要的所有旋转.

First, you don't want to use gluLookAt. gluLookAt rotates the camera, but the physical screen the user looks at doesn't rotate. gluLookAt would only work if the screen would rotate such that the screen normal would keep pointing at the user. The perspective distortion of the off-axis projection will take care of all the rotation we need.

您需要在模型中考虑屏幕在截锥体中的位置.考虑下图.红点是屏幕边框.您需要实现的是这些位置在 3D WCS 中保持不变，因为现实世界中的物理屏幕也(希望)不会移动.我认为这是对虚拟现实和立体视觉的关键洞察.屏幕就像是进入虚拟现实的窗口，要将现实世界与虚拟现实对齐，您需要将视锥体与该窗口对齐.

What you need to factor into your model is the position of the screen within the frustum. Consider the following image. The red points are the screen borders. What you need to achieve is that these positions remain constant in the 3D WCS, since the physical screen in the real world also (hopefully) doesn't move. I think this is the key insight to virtual reality and stereoscopy. The screen is something like a window into the virtual reality, and to align the real world with the virtual reality, you need to align the frustum with that window.

为此，您必须确定屏幕在 Kinect 坐标系中的位置.假设 Kinect 在屏幕顶部，+y 指向下方，并且您使用的单位是毫米，我希望这些坐标沿着 (+-300, 200, 0), (+-300, 500, 0).

To do that you have to determine the position of the screen in the coordinate system of the Kinect. Assuming the Kinect is on top of the screen, that +y points downwards, and that the unit you're using is millimeters, I would expect these coordinates to be something along the lines of (+-300, 200, 0), (+-300, 500, 0).

现在远平面有两种可能性.您可以选择使用从相机到远平面的固定距离.这意味着如果用户向后移动，远平面将向后移动，可能会剪裁您想要绘制的对象.或者，您可以将远平面保持在 WCS 中的固定位置，如图所示.我相信后者更有用.对于近平面，我认为与相机保持固定距离是可以的.

Now there are two possibilities for the far plane. You could either choose to use a fixed distance from the camera to the far plane. That would mean the far plane would move backwards if the user moved backwards, possibly clipping objects you'd like to draw. Or you could keep the far plane at a fixed position in the WCS, as shown in the image. I believe the latter is more useful. For the near plane, I think a fixed distance from the camera is ok though.

输入是屏幕的 3D 位置 wcsPtTopLeftScreen 和 wcsPtBottomRightScreen，头部的跟踪位置 wcsPtHead，z 值远平面 wcsZFar(均在 WCS 中)，以及近平面 camZNear 的 z 值(在相机坐标中).我们需要计算相机坐标中的视锥体参数.

The inputs are the 3D positions of the screen wcsPtTopLeftScreen and wcsPtBottomRightScreen, the tracked position of the head wcsPtHead, the z value of the far plane wcsZFar (all in the WCS), and the z value of the near plane camZNear (in camera coordinates). We need to compute the frustum parameters in camera coordinates.

camPtTopLeftScreen = wcsPtTopLeftScreen - wcsPtHead;
camPtTopLeftNear = camPtTopLeftScreen / camPtTopLeftScreen.z * camZNear;

与右下角相同.还有:

and the same with the bottom right point. Also:

camZFar = wcsZFar - wcsPtHead.z

现在唯一的问题是 Kinect 和 OpenGL 使用不同的坐标系.在 Kinect CS 中，+y 指向下方，+z 从用户指向 Kinect.在 OpenGL 中，+y 指向上方，+z 指向观察者.这意味着我们必须将 y 和 z 乘以 -1:

Now the only problem is that the Kinect and OpenGL use different coordinate systems. In the Kinect CS, +y points down, +z points from the user towards the Kinect. In OpenGL, +y points up, +z points towards the viewer. That means we have to multiply y and z by -1:

glFrustum(camPtTopLeftNear.x, camPtBottomRightNear.x,
  -camPtBottomRightNear.y, -camPtTopLeftNear.y, camZNear, camZFar);

如果您想要更好的解释，也包括立体视觉，请查看此视频，我发现它很有见地并且做得很好.

If you want a better explanation that also covers stereoscopy, check out this video, I found it insightful and well done.

快速演示，您可能需要调整wcsWidth、pxWidth 和wcsPtHead.z.

Quick demo, you might have to adjust wcsWidth, pxWidth, and wcsPtHead.z.

#include <glm/glm.hpp>
#include <glm/ext.hpp>
#include <glut.h>
#include <functional>

float heightFromWidth;
glm::vec3 camPtTopLeftNear, camPtBottomRightNear;
float camZNear, camZFar;
glm::vec3 wcsPtHead(0, 0, -700);

void moveCameraXY(int pxPosX, int pxPosY)
{
  // Width of the screen in mm and in pixels.
  float wcsWidth = 520.0;
  float pxWidth = 1920.0f;

  float wcsHeight = heightFromWidth * wcsWidth;
  float pxHeight = heightFromWidth * pxWidth;
  float wcsFromPx = wcsWidth / pxWidth;

  glm::vec3 wcsPtTopLeftScreen(-wcsWidth/2.f, -wcsHeight/2.f, 0);
  glm::vec3 wcsPtBottomRightScreen(wcsWidth/2.f, wcsHeight/2.f, 0);
  wcsPtHead = glm::vec3(wcsFromPx * float(pxPosX - pxWidth / 2), wcsFromPx * float(pxPosY - pxHeight * 0.5f), wcsPtHead.z);
  camZNear = 1.0;
  float wcsZFar = 500;

  glm::vec3 camPtTopLeftScreen = wcsPtTopLeftScreen - wcsPtHead;
  camPtTopLeftNear = camZNear / camPtTopLeftScreen.z * camPtTopLeftScreen;
  glm::vec3 camPtBottomRightScreen = wcsPtBottomRightScreen - wcsPtHead;
  camPtBottomRightNear = camPtBottomRightScreen / camPtBottomRightScreen.z * camZNear;
  camZFar = wcsZFar - wcsPtHead.z;

  glutPostRedisplay();
}

void moveCameraZ(int button, int state, int x, int y)
{
  // No mouse wheel in GLUT. :(
  if ((button == 0) || (button == 2))
  {
    if (state == GLUT_DOWN)
      return;
    wcsPtHead.z += (button == 0 ? -1 : 1) * 100;
    glutPostRedisplay();
  }
}

void reshape(int w, int h)
{
  heightFromWidth = float(h) / float(w);
  glViewport(0, 0, w, h);
}

void drawObject(std::function<void(GLdouble)> drawSolid, std::function<void(GLdouble)> drawWireframe, GLdouble size)
{
  glPushAttrib(GL_ALL_ATTRIB_BITS);
  glEnable(GL_COLOR);
  glDisable(GL_LIGHTING);
  glColor4f(1, 1, 1, 1);
  drawSolid(size);
  glColor4f(0.8, 0.8, 0.8, 1);
  glDisable(GL_DEPTH_TEST);
  glLineWidth(1);
  drawWireframe(size);

  glColor4f(0, 0, 0, 1);
  glEnable(GL_DEPTH_TEST);
  glLineWidth(3);
  drawWireframe(size);
  glPopAttrib();
}

void display(void)
{
  glPushAttrib(GL_ALL_ATTRIB_BITS);
  glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT);
  glEnable(GL_DEPTH_TEST);

  // In the Kinect CS, +y points down, +z points from the user towards the Kinect.
  // In OpenGL, +y points up, +z points towards the viewer.
  glm::mat4 mvpCube;
  mvpCube = glm::frustum(camPtTopLeftNear.x, camPtBottomRightNear.x,
    -camPtBottomRightNear.y, -camPtTopLeftNear.y, camZNear, camZFar);
  mvpCube = glm::scale(mvpCube, glm::vec3(1, -1, -1));
  mvpCube = glm::translate(mvpCube, -wcsPtHead);
  glMatrixMode(GL_MODELVIEW); glLoadMatrixf(glm::value_ptr(mvpCube));

  drawObject(glutSolidCube, glutWireCube, 140);

  glm::mat4 mvpTeapot = glm::translate(mvpCube, glm::vec3(100, 0, 200));
  mvpTeapot = glm::scale(mvpTeapot, glm::vec3(1, -1, -1)); // teapots are in OpenGL coordinates
  glLoadMatrixf(glm::value_ptr(mvpTeapot));
  glColor4f(1, 1, 1, 1);
  drawObject(glutSolidTeapot, glutWireTeapot, 50);

  glFlush();
  glPopAttrib();
}

void leave(unsigned char, int, int)
{
  exit(0);
}

int main(int argc, char **argv)
{
  glutInit(&argc, argv);
  glutCreateWindow("glut test");
  glutDisplayFunc(display);
  glutReshapeFunc(reshape);
  moveCameraXY(0,0);
  glutPassiveMotionFunc(moveCameraXY);
  glutMouseFunc(moveCameraZ);
  glutKeyboardFunc(leave);
  glutFullScreen();
  glutMainLoop();
  return 0;
}

以下图片的观看距离应为屏幕宽度的 135%(在我的 52 厘米宽屏幕上全屏显示为 70 厘米).

The following images should be viewed from a distance equal to 135% of their width on screen (70 cm on my 52 cm wide screen in fullscreen).

这篇关于用于 OpenGL 中头部跟踪的斜视锥体/离轴投影的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！