This forum is disabled, please visit https://forum.opencv.org
 | 1 | initial version |
Well, I'm never worked with Kinect 2, but I suppose that it is very similar to the original Kinect. The fact is that the Kinect has 2 cameras (one IR and another RGB), and it generates it depth map by projecting a pattern over the scene using a IR projector and computing the depths by using the input of the IR camera.
That said, the raw data from the RGB camera wont be aligned with the depth data because the cameras are distinct. It is not a mere matter of just resizing the images. The depth map has to be properly mapped , or better yet, registrated. This means that you have to re-project the depth points into the RGB camera point of view and assign the color
As I said, I'm not sure how you capture the data from the Kinect 2. If you are using the OpenNI backend of OpenCV, you can do just as I explained in this another question HERE, and the depth map will be properly registered with the RGB image.
If the way that you are extracting this data from the Kinect is not this, you must determine the camera pose of the RGB camera relative to the IR camera pose to do the reprojection.
Calibrating the cameras (only once !):
Calibrate both cameras as a stereo pair, using cv::stereoCalibrate with the camera matrices that you computed in the previous step and a calibration pattern viewed by the same cameras. You will obtain the rotation, translation needed to re-project the points. If you have the camera matrices of the RGB and IR camera a priori, it will be much better. The ones from Kinetic 1 are quite well known and can be found in internet. I believe that it is simple to find the ones from Kinetic 2 at this point...
Colouring the depth map (for every frame):
With the procedure above, you have the rotation and translation matrices and the camera matrix of the RGB camera. All that you have to do is project the 3D depth point into the RGB camera space using cv::projectPoints, and assign the colour of the pixel (or interpolation of the nearby pixels). Note that some points will be colourless since they aren't visible by the RGB camera.
