使用相邻像素的叉积从深度图像计算表面法线 [英] Calculate surface normals from depth image using neighboring pixels cross product

问题描述

正如标题所述，我想通过使用相邻像素的叉积来计算给定深度图像的表面法线.我想为此使用Opencv并避免使用PCL，但是我并不真正理解该过程，因为我对该学科的了解非常有限.因此，如果有人可以提供一些提示，我将不胜感激.这里要说的是，除了深度图像和相应的rgb图像之外，我没有其他信息，因此没有K相机矩阵信息.

As the title says I want to calculate the surface normals of a given depth image by using the cross product of neighboring pixels. I would like to use Opencv for that and avoid using PCL however, I do not really understand the procedure, since my knowledge is quite limited in the subject. Therefore, I would be grateful is someone could provide some hints. To mention here that I do not have any other information except the depth image and the corresponding rgb image, so no K camera matrix information.

因此，可以说我们有以下深度图像:

Thus, lets say that we have the following depth image:

我想在对应点找到具有相应深度值的法线向量，如下图所示:

and I want to find the normal vector at a corresponding point with a corresponding depth value like in the following image:

如何使用相邻像素的叉积来实现?我不介意法线不是很准确.

How can I do that using the cross product of the neighbouring pixels? I do not mind if the normals are not highly accurate.

谢谢.

更新:

好吧，我试图遵循@timday的回答并将其代码移植到Opencv.使用以下代码:

Ok, I was trying to follow @timday's answer and port his code to Opencv. With the following code:

Mat depth = <my_depth_image> of type CV_32FC1
Mat normals(depth.size(), CV_32FC3);

for(int x = 0; x < depth.rows; ++x)
{
    for(int y = 0; y < depth.cols; ++y)
    {

        float dzdx = (depth.at<float>(x+1, y) - depth.at<float>(x-1, y)) / 2.0;
        float dzdy = (depth.at<float>(x, y+1) - depth.at<float>(x, y-1)) / 2.0;

        Vec3f d(-dzdx, -dzdy, 1.0f);
        Vec3f n = normalize(d);

        normals.at<Vec3f>(x, y) = n;
    }
}

imshow("depth", depth / 255);
imshow("normals", normals);

我得到了正确的以下结果(我不得不用float和Vecd替换double和Vecf，但是我不知道为什么会有什么不同):

I am getting the correct following result (I had to replace double with float and Vecd to Vecf, I do not know why that would make any difference though):

推荐答案

您真的不需要 为此使用叉积，但请参见下文.

You don't really need to use the cross product for this, but see below.

考虑您的范围图像是一个函数z(x，y).

Consider your range image is a function z(x,y).

表面法线在(-dz/dx，-dz/dy，1)方向上. (这里的dz/dx是指微分:z随x的变化率).然后按照常规将法线标准化为单位长度.

The normal to the surface is in the direction (-dz/dx,-dz/dy,1). (Where by dz/dx I mean the differential: the rate of change of z with x). And then normals are conventionally normalized to unit length.

顺便说一句，如果您想知道(-dz/dx，-dz/dy，1)的来源...如果您将平面平行线中的两个正交切向量取到x和y轴，则这些是(1,0，dzdx)和(0,1，dzdy).法线垂直于切线，因此法线应为(1,0，dzdx)X(0,1，dzdy)-其中'X'是叉积-(-dzdx，-dzdy，1).因此，有一个叉积派生的法线，但是只需直接将结果表达式用作法线，就无需在代码中进行显式计算.

Incidentally, if you're wondering where that (-dz/dx,-dz/dy,1) comes from... if you take the 2 orthogonal tangent vectors in the plane parellel to the x and y axes, those are (1,0,dzdx) and (0,1,dzdy). The normal is perpendicular to the tangents, so should be (1,0,dzdx)X(0,1,dzdy) - where 'X' is cross-product - which is (-dzdx,-dzdy,1). So there's your cross product derived normal, but there's little need to compute it so explicitly in code when you can just use the resulting expression for the normal directly.

用于在(x，y)处计算单位长度法线的伪代码将类似于

Pseudocode to compute a unit-length normal at (x,y) would be something like

dzdx=(z(x+1,y)-z(x-1,y))/2.0;
dzdy=(z(x,y+1)-z(x,y-1))/2.0;
direction=(-dzdx,-dzdy,1.0)
magnitude=sqrt(direction.x**2 + direction.y**2 + direction.z**2)
normal=direction/magnitude

根据您要执行的操作，用较大的数字替换NaN值可能更有意义.

Depending on what you're trying to do, it might make more sense to replace the NaN values with just some large number.

使用这种方法，从您的范围图像中，我可以得到:

Using that approach, from your range image, I can get this:

(然后，我使用计算出的法线方向进行一些简单的着色；由于距离图像的量化，请注意"steppy"外观；理想情况下，对于真实距离数据，精度要高于8位).

(I'm then using the normal directions calculated to do some simple shading; note the "steppy" appearance due to the range image's quantization; ideally you'd have higher precision than 8-bit for the real range data).

对不起，不是OpenCV或C ++代码，只是为了完整性:下面是产生该图像的完整代码(GLSL嵌入Qt QML文件；可以与Qt5的qmlscene一起运行).上面的伪代码可以在片段着色器的main()函数中找到:

Sorry, not OpenCV or C++ code, but just for completeness: the complete code which produced that image (GLSL embedded in a Qt QML file; can be run with Qt5's qmlscene) is below. The pseudocode above can be found in the fragment shader's main() function:

import QtQuick 2.2

Image {
  source: 'range.png'  // The provided image

  ShaderEffect {
    anchors.fill: parent
    blending: false

    property real dx: 1.0/parent.width
    property real dy: 1.0/parent.height
    property variant src: parent

    vertexShader: "
      uniform highp mat4 qt_Matrix;
      attribute highp vec4 qt_Vertex;
      attribute highp vec2 qt_MultiTexCoord0;
      varying highp vec2 coord;
      void main() {
        coord=qt_MultiTexCoord0;
        gl_Position=qt_Matrix*qt_Vertex;
      }"

   fragmentShader: "
     uniform highp float dx;
     uniform highp float dy;
     varying highp vec2 coord;
     uniform sampler2D src;
     void main() {
       highp float dzdx=( texture2D(src,coord+vec2(dx,0.0)).x - texture2D(src,coord+vec2(-dx,0.0)).x )/(2.0*dx);
       highp float dzdy=( texture2D(src,coord+vec2(0.0,dy)).x - texture2D(src,coord+vec2(0.0,-dy)).x )/(2.0*dy);
       highp vec3 d=vec3(-dzdx,-dzdy,1.0);
       highp vec3 n=normalize(d);
       highp vec3 lightDirection=vec3(1.0,-2.0,3.0);
       highp float shading=0.5+0.5*dot(n,normalize(lightDirection));
       gl_FragColor=vec4(shading,shading,shading,1.0);
     }"
  }
}

这篇关于使用相邻像素的叉积从深度图像计算表面法线的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！