Ask Your Question
1

Kinect RGB and depth frame

asked 2015-05-28 20:01:33 -0500

dmngu9 gravatar image

updated 2015-05-28 20:08:06 -0500

I have RGB and depth frame both have resolution of 480x640 pixels. I already align these 2 frames together and what i obtained is the pointcloud rgb-d . How can i implement SURF with these RGB-D image since SURF opencv only for 2d image?

I am also confused about this situation. If i have a point in RGB image, lets say (100,200). Now i want to find the depth of that point. Can i just go to the point (100,200) in depth registered image and get the depth out?

I use ROS Kinect OpenNI

Please let me know

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
2

answered 2015-05-29 04:31:49 -0500

R.Saracchini gravatar image

updated 2015-06-01 02:20:50 -0500

The registered depth data is given in a cv::Mat structure with data in several types. If you retrieve it by using CV_CAP_OPENNI_DEPTH_MAP or CV_CAP_OPENNI_POINT_CLOUD_MAP, this matrix will be respectively of CV_16UC1 or CV_32FC3. The first is a 16-bit integer with the depth in millimetres, and the second each element will be an X,Y,Z vector of the relative 3D position of the point to the camera in metres . You can easily access the data by accessing the individual elements of the Mat object.

see the example:

cv:.Mat image3D, imageBGR;
cv::VideoCapture videoReader;
//Init camera
videoReader.open( CV_CAP_OPENNI );
videoReader.set( CV_CAP_OPENNI_IMAGE_GENERATOR_OUTPUT_MODE, CV_CAP_OPENNI_VGA_30HZ );
videoReader.set(CV_CAP_OPENNI_DEPTH_GENERATOR_REGISTRATION,1); 
//grab a frame
if (videoReader.grab()){
    videoReader.retrieve(imageBGR,CV_CAP_OPENNI_BGR_IMAGE);
    videoReader.retrieve(image3D,CV_CAP_OPENNI_POINT_CLOUD_MAP);
   //Here you get the depth
   int y = 200;
   int x = 100;
  cv::Vec3f pt_3d = image3D.at<cv::Vec3f>(y,x); //0 - X, 1 - Y, 2 - Z
else{
  std::cerr << "Error capturing frame !" << std::endl;
}

I hope that it helps.

About the alignment of the images:

The flags sent in the "set" command to the "VideoCapture " structure in the example above (more specifically "CV_CAP_OPENNI_DEPTH_GENERATOR_REGISTRATION") ensure that the RGB and the depth frame will be properly registered/aligned by the OpenNI backend. As side effect of the registration, there will be a black border around the scene which corresponds to regions where the field of view of the IR sensor does not overlap with the RGB sensor.

See the example bellow. As you can see, there is a surrounding area where there isn't any depth information (the black regions):

image description image description

edit flag offensive delete link more

Comments

thanks for your help. but my question is that 2 frames rgb and depth both have same resolution. if i have point (u,v) in rgb frame, is the point (u,v) in registered depth map corresponding to the point(u,v) in rgb frame (in other words aligning with each other ).

dmngu9 gravatar imagedmngu9 ( 2015-05-30 06:47:45 -0500 )edit
1

I have edited the answer in order to clarify better your doubts.

R.Saracchini gravatar imageR.Saracchini ( 2015-06-01 02:21:39 -0500 )edit

Hey, piggybacking this: Do you know how OpenNI generates the point cloud? It's hard to find documentation about it. What is the origin of the coordinate system? Is it the IR Sensor? How does OpenNI generate the X and Y values in meters? There has to be some sort of internal calibration, to calculate (X,Y)meters from (X,Y)pixels+(Z)meters. I can't use the point cloud feature in my bachelors thesis unless I know how it works. Thanks in Advance :)

CaptainSwagpants gravatar imageCaptainSwagpants ( 2015-08-19 08:44:52 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2015-05-28 20:01:33 -0500

Seen: 1,795 times

Last updated: Jun 01 '15