对齐已经捕获的rgb和深度图像 [英] Align already captured rgb and depth images

问题描述

我正在尝试使用MATLAB对齐两个图像 - 一个rgb和另一个深度。请注意，我已经检查了几个地方 - 比如

基本事实基本上意味着已经提供了指定感兴趣的面部区域的边界框，并且我仅使用它们来裁剪面部区域。 matlab代码如下所示：

  I = imread（'1.jpg'）; 
 I1 = imcrop（I，[218,198,158,122]）; 
 I2 = imcrop（I，[243,209,140,​​108]）; 
 figure，subplot（1,2,1），imshow（I1）; 
 subplot（1,2,2），imshow（I2）; 
  

两个裁剪的图像rgb和深度如下所示：

---

我们可以通过哪种方式注册/分配图像。我从
。

然而，它们是如此嘈杂，我不认为我们将能够为这些图像进行关键点匹配。

有人可以在matlab中建议一些算法做同样的事吗？

解决方案

序言

此答案基于我以前的答案：

• 
  
  

  所以我的结果受到所有人的影响这是消极的。无论如何，这就是你的意思ed to do：

  
  

  1. 确定两张图片的FOV

     
     
     

     因此，在两个图像上都可以看到一些可测量的特征。尺寸越大，结果越准确。例如，我选择这些：

     
     
     

  
  

  2. 对齐

     
     
     

     如果您注意到我将 ofsx，ofsy 变量添加到 copy_images（）。这是相机之间的偏移。我通过 1 像素在箭头按键上更改它们，然后调用 copy_images 并渲染结果。这样我就可以非常快速地手动找到偏移量：

     
     
     
     
     
     

     As你可以看到偏移是 +17 x轴上的像素和y轴上的 +4 像素。这里是侧视图，以便更好地了解深度：

     
     
     

  
  

  希望它有点帮助

  
  

  I am trying to allign two images - one rgb and another depth using MATLAB. Please note that I have checked several places for this - like here , here which requires a kinect device, and here here which says that camera parameters are required for calibration. I was also suggested to use EPIPOLAR GEOMETRY to match the two images though I do not know how. The dataset I am referring to is given in rgb-d-t face dataset. One such example is illustrated below :

  The ground truth which basically means the bounding boxes which specify the face region of interest are already provided and I use them to crop the face regions only. The matlab code is illustrated below :

  I = imread('1.jpg');
  I1 = imcrop(I,[218,198,158,122]);
  I2 = imcrop(I,[243,209,140,108]);
  figure, subplot(1,2,1),imshow(I1);
  subplot(1,2,2),imshow(I2);
  

  The two cropped images rgb and depth are shown below :

  ---

  Is there any way by which we can register/allign the images. I took the hint from here where basic sobel operator has been used on both the rgb and depth images to generate an edge map and then keypoints will need to be generated for matching purposes. The edge maps for both the images are generated here.

  .

  However they are so noisy that I do not think we will be able to do keypoint matching for this images.

  Can anybody suggest some algorithms in matlab to do the same ?

  解决方案

  prologue

  This answer is based on mine previous answer:

  • Does Kinect Infrared View Have an offset with the Kinect Depth View

  I manually crop your input image so I separate colors and depth images (as my program need them separated. This could cause minor offset change by few pixels. Also as I do not have the depths (depth image is 8bit only due to grayscale RGB) then the depth accuracy I work with is very poor see:

  So my results are affected by all this negatively. Anyway here is what you need to do:

  1. determine FOV for both images

     So find some measurable feature visible on both images. The bigger in size the more accurate the result. For example I choose these:

  2. form a point cloud or mesh

     I use depth image as reference so my point cloud is in its FOV. As I do not have the distances but 8bit values instead I converted that to some distance by multiplying by constant. So I scan whole depth image and for every pixel I create point in my point cloud array. Then convert the dept pixel coordinate to color image FOV and copy its color too. something like this (in C++):

     picture rgb,zed; // your input images
     struct pnt3d { float pos[3]; DWORD rgb; pnt3d(){}; pnt3d(pnt3d& a){ *this=a; }; ~pnt3d(){}; pnt3d* operator = (const pnt3d *a) { *this=*a; return this; }; /*pnt3d* operator = (const pnt3d &a) { ...copy... return this; };*/ };
     pnt3d **xyz=NULL; int xs,ys,ofsx=0,ofsy=0;
     
     void copy_images()
         {
         int x,y,x0,y0;
         float xx,yy;
         pnt3d *p;
         for (y=0;y<ys;y++)
          for (x=0;x<xs;x++)
             {
             p=&xyz[y][x];
             // copy point from depth image
             p->pos[0]=2.000*((float(x)/float(xs))-0.5);
             p->pos[1]=2.000*((float(y)/float(ys))-0.5)*(float(ys)/float(xs));
             p->pos[2]=10.0*float(DWORD(zed.p[y][x].db[0]))/255.0;
             // convert dept image x,y to color image space (FOV correction)
             xx=float(x)-(0.5*float(xs));
             yy=float(y)-(0.5*float(ys));
             xx*=98.0/108.0;
             yy*=106.0/119.0;
             xx+=0.5*float(rgb.xs);
             yy+=0.5*float(rgb.ys);
             x0=xx; x0+=ofsx;
             y0=yy; y0+=ofsy;
             // copy color from rgb image if in range
             p->rgb=0x00000000; // black
             if ((x0>=0)&&(x0<rgb.xs))
              if ((y0>=0)&&(y0<rgb.ys))
               p->rgb=rgb2bgr(rgb.p[y0][x0].dd); // OpenGL has reverse RGBorder then my image
             }
         }
     

     where **xyz is my point cloud 2D array allocated t depth image resolution. The picture is my image class for DIP so here some relevant members:

     • xs,ys is the image resolution in pixels
     • p[ys][xs] is the image direct pixel access as union of DWORD dd; BYTE db[4]; so I can access color as single 32 bit variable or each color channel separately.
     • rgb2bgr(DWORD col) just reorder color channels from RGB to BGR.

  3. render it

     I use OpenGL for this so here the code:

         glBegin(GL_QUADS);
         for (int y0=0,y1=1;y1<ys;y0++,y1++)
         for (int x0=0,x1=1;x1<xs;x0++,x1++)
             {
             float z,z0,z1;
             z=xyz[y0][x0].pos[2]; z0=z; z1=z0;
             z=xyz[y0][x1].pos[2]; if (z0>z) z0=z; if (z1<z) z1=z;
             z=xyz[y1][x0].pos[2]; if (z0>z) z0=z; if (z1<z) z1=z;
             z=xyz[y1][x1].pos[2]; if (z0>z) z0=z; if (z1<z) z1=z;
             if (z0   <=0.01) continue;
             if (z1   >=3.90) continue;  // 3.972 pre vsetko nad .=3.95m a 4.000 ak nechyti vobec nic
             if (z1-z0>=0.10) continue;
             glColor4ubv((BYTE* )&xyz[y0][x0].rgb);
             glVertex3fv((float*)&xyz[y0][x0].pos);
             glColor4ubv((BYTE* )&xyz[y0][x1].rgb);
             glVertex3fv((float*)&xyz[y0][x1].pos);
             glColor4ubv((BYTE* )&xyz[y1][x1].rgb);
             glVertex3fv((float*)&xyz[y1][x1].pos);
             glColor4ubv((BYTE* )&xyz[y1][x0].rgb);
             glVertex3fv((float*)&xyz[y1][x0].pos);
             }
         glEnd();
     

     You need to add the OpenGL initialization and camera settings etc of coarse. Here the unaligned result:

  4. align it

     If you notice I added ofsx,ofsy variables to copy_images(). This is the offset between cameras. I change them on arrows keystrokes by 1 pixel and then call copy_images and render the result. This way I manually found the offset very quickly:

     As you can see the offset is +17 pixels in x axis and +4 pixels in y axis. Here side view to better see the depths:

  Hope It helps a bit

  这篇关于对齐已经捕获的rgb和深度图像的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！