对齐捕获的深度和RGB图像 [英] Aligning captured depth and rgb images
问题描述
以前曾有过问题(此处，此处)与我的问题有关，但是我的问题与我的问题有不同的方面，这在我之前的任何文章中都没有见过问的问题.
There has been previous questions (here, here and here) related to my question, however my question has a different aspect to it, which I have not seen in any of the previously asked questions.
我已经使用Kinect深度传感器为我的研究获取了一个数据集.该数据集在特定瞬间针对深度和rgb流均采用.png图像格式.为了给您更多的想法，下面是这些框架:
I have acquired a dataset for my research using Kinect Depth sensor. This dataset is in the format of .png images for both depth and rgb stream at a specific instant. To give you more idea below are the frames:
我在这里添加边缘检测输出.
I am adding the edge detection output here.
Sobel Edge检测输出用于:
Sobel Edge detection output for:
-
RGB图像
RGB Image
深度图像
现在我要做的是将这两个帧对齐以给我合成RGBZ图像.
Now what I am trying to do is align these two frames to give me a combined RGBZ image.
我不了解基本的摄像机特性，也不知道rgb和红外传感器之间的距离.
I do not have knowledge of the underlying camera characteristics or the distance between both rgb and infrared sensors.
有没有一种方法可以将RGB值与相应的Z值进行匹配?
Is there a method which can be applied to match the RGB values to the corresponding Z values?
我的想法之一是在两个图像中都使用边缘，并尝试使其匹配.
One of the ideas I have is to use edges in both images and try to match them.
推荐答案
通常，您试图从RGB和深度图像对中进行的操作是不平凡且定义不明确的.作为人类，我们可以在RGB图像中识别出手臂，并且能够将其与深度图像的更靠近相机的区域相关联.但是，计算机没有关于其期望将RGB图像的哪些部分对应于深度图像的哪些部分的先验知识.
In general what you are trying to do from a pair of RGB and Depth images is non-trivial and ill-defined. As humans we recognise the arm in the RGB image, and are able to relate it to the area of the depth image closer to the camera. However, a computer has no prior knowledge about which parts of the RGB image it expects to correspond to which parts of the depth image.
大多数用于这种对准的算法都使用相机校准的原因是，此过程使此不适定的问题变得正确.
The reason most algorithms for such alignment use camera calibration is that this process allows this ill-posed problem to become well-posed.
但是，仍然可能存在找到对应关系的方法，特别是如果您有大量来自同一Kinect的图像对时.然后，您仅需要搜索一组转换参数.我不知道有任何现有算法可以执行此操作，但是正如您在问题中所指出的那样，您可能会发现在两个图像上都进行边缘检测并尝试将边缘图像对齐是一个不错的起点.
However, there may still be ways to find the correspondences, particularly if you have lots of image pairs from the same Kinect. You then need only search for one set of transformation parameters. I don't know of any existing algorithms to do this, but as you note in your question you may find something like doing edge detection on both images and trying to align the edge images a good place to start.
最后，请注意，当物体靠近Kinect时，即使在校正图像后，RGB和深度图像之间的对应关系也会变差.您可以在图像中看到一些这种效果-手在示例深度图像中产生的阴影"在某种程度上表明了这一点.
Finally, note that when objects get close to the Kinect the correspondence between RGB and depth images can become poor, even after the images have been calibrated. You can see some of this effect in your images - the 'shadow' that the hand makes in your example depth image is somewhat indicative of this.
这篇关于对齐捕获的深度和RGB图像的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
