Transfer kinect depth data from c++ vector to Mat and display it in OpenCV

asked 2018-12-11 15:18:08 -0600

Ed95
11 ●1 ●3

updated 2018-12-14 23:18:10 -0600

Hi,

I'm trying to learn OpenCV using a kinect v1 sensor.

OpenCV 4 and libfreenect are installed on ubuntu and I can run the sample programs.

My aim is to store the incoming images from the kinect in a cv::Mat structure and then display them.

I'm trying to adapt the cppview sample program in this libfreenect library.

All the Opengl/GLUT display code was removed and I added the function convertFrames() in an attempt to initialise the Mat data structures with the depth and RGB buffers. I then tried to display the two Mat structures using imshow()

A blank 'Image' frame is displayed but then closes. I'd appreciate any help on how to initialisation/populate the Mat structures correctly.

Here's the code that attempts to populate the Mat data structures and then display them:

     //convert the frame buffer to opencv Mat
    void convertFrames()
    {
    static std::vector<uint8_t> depth(640*480*4);
    static std::vector<uint8_t> rgb(640*480*4);

    printf("Vector depth size = %d ",depth.size());
    device->updateState();
    //  printf("\r demanded tilt angle: %+4.2f device tilt angle: %+4.2f", freenect_angle, device->getState().getTiltDegs());
    fflush(stdout);

    device->getDepth(depth);
    device->getRGB(rgb);
    Mat mRGB_tmp (640, 480, CV_8UC3,&rgb);
    Mat mDepth_tmp (640, 480, CV_16UC1,&depth);
    cout<<"Depth ="<< mDepth_tmp << endl;

    imshow("Image",mRGB_tmp);
    imshow("Depth",mDepth_tmp);
    got_frames = 0;

}

/*UPDATE*/

Thanks to sjhalayka and berak. Have used the correct clone() declaration, updated Opencv and used waitkey() and can now use imshow() to output the depth image but there's an issue with an overlap.

The image seems to be overlaid with an offset. Here's a screenshot of the top of a travel bag, with handle extended. I discovered the resolution image size of the depth camera on the v1 is 320x240, - the buffers are 640x480 - but this code is from the sample code which worked (usinf OpenGL instead of OpenCV's imshow()).

As I understand the original code a depth buffer is created using <uint8_t>, even though the depth data is 16 bits. This is compensated by 'doubling the data type size to 4 ( 640x480x4). The RGB buffer is the same size (assume it is one byte for each color then 1 byte for gamma channel). The data is copied from the callback buffer, which is <uint16_t>., which is then copied into a cv::Mat <CV_16UC1> buffer and then displayed.

I'm a bit confused by the overlay/offset error and how the 320x240 image is being handled as a 640x480. Any help is appreciated.

edit retag flag offensive close merge delete

Comments

Can you please pare down the code, so that only the code in question is given?

sjhalayka ( 2018-12-11 17:54:16 -0600 )edit

@sjhalayka . Just included the .cpp file at the end for reference but have removed. thx

Ed95 ( 2018-12-11 18:16:04 -0600 )edit

I'm not really sure what the problem is. The following code shows how to use the address of the vector's first element, to make a 640x480 16-bit single channel image:

Mat frame_content = Mat(480, 640, CV_16UC1, &depths[0]).clone();

sjhalayka ( 2018-12-11 21:31:53 -0600 )edit

like mentioned above, you got width and height wrong, and the address of a vector is NOT the address of the 1st element. either use &vec[0] or vec.data()

berak ( 2018-12-12 00:17:58 -0600 )edit

@sjhalayka Thanks.I had the rows and columns around the wrong way. If I use clone() I now get an exception when using imshow()

 terminate called after throwing an instance of 'cv::Exception'what():  OpenCV(4.0.0) /home/ed/src/opencv-4.0.0/modules/highgui/src/window_gtk.cpp:146: error: (-215:Assertion failed) dst.data == widget->original_image->data.ptr in function 'cvImageWidgetSetImage'

From what I've read the depth values for kinect v1 are between 0-2047. The documentation for imshow() mentioned that if you use AUTOSIZE when you create the window it will be displayed @ original size. Do I need to do any conversion between CV16UC1 to CV8UC1 before calling imshow()?

Ed95 ( 2018-12-12 02:13:58 -0600 )edit

@berak. Thanks, that was the first thing I saw too and that's a great ptr explanation!

Ed95 ( 2018-12-12 02:18:30 -0600 )edit

@Ed95 the last one looks like a recent bug. (imshow can only show 8bit data). it was fixed in the meantime, but you need to update your opencv libs to latest master & rebuild.

in the meantime, convert your depth image manually to 8U before using imshow:

depthimg.convertTo(depthimg, CV_8U, 1.0/255);
imshow("depth", depthimg);

berak ( 2018-12-12 02:59:54 -0600 )edit

@berak. Thanks that was perfect. Converting to CV_8U fixed the exception.Will update libs. The window isn't appearing or being updated though. The code I have for displaying the frames is something like :

   cv::namedWindow("Depth", WINDOW_AUTOSIZE);  //in the main
   depthimg.convertTo(depthimg, CV_8U, 1.0/255);       // in the function
   imshow("Depth", depthimg);                                //in the function

No doubt something simple. Will look into it.

Ed95 ( 2018-12-12 03:58:07 -0600 )edit

here's the issue

berak ( 2018-12-12 04:00:09 -0600 )edit

I read that you need to use waitKey() with a value greater than 0, and the depth image resolution for v1 is only 320x240. Implementing waitkey() I now get a black screen. If I play with the buffer sizes I can get a low quality image/mirrored image, with some strange overlay when the kinect is moved/rotated.Think now it's an issue with buffer sizes and scaling issues.

static std::vector<uint16_t> kdepth( 640 * 480 * 4); 
static unsigned int dBufSize = 640 * 480 * 4;
int dWidth = 320;
int dHeight = 240;
Mat mDepth = Mat(dHeight, dWidth, CV_16UC1,&depth[0]).clone();

mDepth.convertTo(mDepth,CV_8U, 1.0/255);
imshow("Depth",mDepth);
waitKey(1);

kinect image of sd card adpater

Ed95 ( 2018-12-12 20:47:00 -0600 )edit

add a comment

Transfer kinect depth data from c++ vector to Mat and display it in OpenCV

Comments

1 answer

Links

Question Tools

Stats

Related questions

Transfer kinect depth data from c++ vector to Mat and display it in OpenCV edit

Comments

1 answer

Links

Question Tools

Stats

Related questions

Transfer kinect depth data from c++ vector to Mat and display it in OpenCV