Kinect one stereo calibration and overlay rgb with depth map

edit

I have recorded some frames (Color and Depth as png type) at 30fps and I didn't saved coordinateMapper. My goal is to post processing these frames and compute 3D points from color map. I understand that the only way to do this is to make a stereoCalibration of the Kinect that I used. Then i'll compute 3D point with depth map and apply rotation and translation matrix.

Did I miss something or this way is a correct way ?

you say you have the Color and Depth frames from the Kinect, I still don't understand why do you want to compute the depth map??? The TOF reconstructed depth map is much better than the one you can obtain with stereoscopic reconstruction.

First of all thank for helping me.

Secondly, I don't want to compute depth map. I just want to get a 3D point for a pixel on the color map. It's just like the MapColorPointToCameraSpacePoint function in coordinatemapper class

If you got the Kinect, you can extract the necessary parameters for the registration using libfreenect2 (Freenect2Device::getIrCameraParams and getColorCameraParams).

Then you can use the Registration::distort and Registration::depth_to_color functions with these parameters (from the registration.cpp).

Thank a lot i'll take a look on libfreenect2 and i'll tell you if it's good

Here are the parameters from my sensor:

IR: fx=365.481 fy=365.481 cx=257.346 cy=210.347 k1=0.089026 k2=-0.271706 k3=0.0982151 p1=0 p2=365.481

Color: fx=1081.37 fy=1081.37 cx=959.5 cy=539.5 shift_d=863 shift_m=52 mx_x3y0=0.000449294 mx_x0y3=1.91656e-05 mx_x2y1=4.82909e-05 mx_x1y2=0.000353673 mx_x2y0=-2.44043e-05 mx_x0y2=-1.19426e-05 mx_x1y1=0.000988431 mx_x1y0=0.642474 mx_x0y1=0.00500649 mx_x0y0=0.142021 my_x3y0=4.42793e-06 my_x0y3=0.000724863 my_x2y1=0.000398557 my_x1y2=4.90383e-05 my_x2y0=0.000136024 my_x0y2=0.00107291 my_x1y1=-1.75465e-05 my_x1y0=-0.00554263 my_x0y1=0.641807 my_x0y0=0.0180811

You might try to copy the functions I mentioned earlier from libfreenect and use them with the parameters above.

@cocs78 sorry for my delayed response. In my case I abandoned the use of the official SDK and I am using the libfreenect2 library as well. Using the libfreenect2 I was able to save the rgb and depth images which were also calibrated from the registration->apply() function. By using this function you can also get straight forward also the RGBXYZ info. However, in my case when I did the recording I was not aware in order to save this information. So, now that I needed to create a point cloud of my data I had to manually extract the camera parameters as @kbarni tells you above and then create the point cloud by using the pcl library. I can provide you this code if you want.

Ok thank you for your reply, I will try it today. Of course, if you could share your code with me, it will be really helpful. How can we proceed ? GitHub ?

@cocs78 with the following you should be fine. Get the depth.h and depth.cpp and create your point cloud as follows:

tools::Camera cameraParams(959.5, 539.5, 1081.37, 1081.37);
cv::Mat mat = cv::imread("color_img");
cv::Mat depth = cv::imread("depth_img", CV_LOAD_IMAGE_ANYDEPTH);

pcl::PointCloud<pcl::PointXYZRGBA>::Ptr pointCloud = tools::Depth::pointCloudFromBGRAndDepth(mat, depth, cameraParams);

pcl::visualization::PCLVisualizer viewer("PCL Viewer");
viewer.setBackgroundColor(0.0, 0.0, 0.5);
viewer.addPointCloud<pcl::PointXYZRGBA>(pointCloud, "point cloud");

while (!viewer.wasStopped()) { viewer.spinOnce (); }