Ask Your Question
0

Transfer kinect depth data from c++ vector to Mat and display it in OpenCV

asked 2018-12-11 15:18:08 -0500

Ed95 gravatar image

updated 2018-12-14 23:18:10 -0500

Hi,

I'm trying to learn OpenCV using a kinect v1 sensor.

OpenCV 4 and libfreenect are installed on ubuntu and I can run the sample programs.

My aim is to store the incoming images from the kinect in a cv::Mat structure and then display them.

I'm trying to adapt the cppview sample program in this libfreenect library.

All the Opengl/GLUT display code was removed and I added the function convertFrames() in an attempt to initialise the Mat data structures with the depth and RGB buffers. I then tried to display the two Mat structures using imshow()

A blank 'Image' frame is displayed but then closes. I'd appreciate any help on how to initialisation/populate the Mat structures correctly.

Here's the code that attempts to populate the Mat data structures and then display them:

     //convert the frame buffer to opencv Mat
    void convertFrames()
    {
    static std::vector<uint8_t> depth(640*480*4);
    static std::vector<uint8_t> rgb(640*480*4);

    printf("Vector depth size = %d ",depth.size());
    device->updateState();
    //  printf("\r demanded tilt angle: %+4.2f device tilt angle: %+4.2f", freenect_angle, device->getState().getTiltDegs());
    fflush(stdout);

    device->getDepth(depth);
    device->getRGB(rgb);
    Mat mRGB_tmp (640, 480, CV_8UC3,&rgb);
    Mat mDepth_tmp (640, 480, CV_16UC1,&depth);
    cout<<"Depth ="<< mDepth_tmp << endl;

    imshow("Image",mRGB_tmp);
    imshow("Depth",mDepth_tmp);
    got_frames = 0;

}

/*UPDATE*/

Thanks to sjhalayka and berak. Have used the correct clone() declaration, updated Opencv and used waitkey() and can now use imshow() to output the depth image but there's an issue with an overlap.

The image seems to be overlaid with an offset. Here's a screenshot of the top of a travel bag, with handle extended. I discovered the resolution image size of the depth camera on the v1 is 320x240, - the buffers are 640x480 - but this code is from the sample code which worked (usinf OpenGL instead of OpenCV's imshow()).

As I understand the original code a depth buffer is created using <uint8_t>, even though the depth data is 16 bits. This is compensated by 'doubling the data type size to 4 ( 640x480x4). The RGB buffer is the same size (assume it is one byte for each color then 1 byte for gamma channel). The data is copied from the callback buffer, which is <uint16_t>., which is then copied into a cv::Mat <CV_16UC1> buffer and then displayed.

I'm a bit confused by the overlay/offset error and how the 320x240 image is being handled as a 640x480. Any help is appreciated.

edit retag flag offensive close merge delete

Comments

Can you please pare down the code, so that only the code in question is given?

sjhalayka gravatar imagesjhalayka ( 2018-12-11 17:54:16 -0500 )edit

@sjhalayka . Just included the .cpp file at the end for reference but have removed. thx

Ed95 gravatar imageEd95 ( 2018-12-11 18:16:04 -0500 )edit
2

I'm not really sure what the problem is. The following code shows how to use the address of the vector's first element, to make a 640x480 16-bit single channel image:

Mat frame_content = Mat(480, 640, CV_16UC1, &depths[0]).clone();
sjhalayka gravatar imagesjhalayka ( 2018-12-11 21:31:53 -0500 )edit
1

like mentioned above, you got width and height wrong, and the address of a vector is NOT the address of the 1st element. either use &vec[0] or vec.data()

berak gravatar imageberak ( 2018-12-12 00:17:58 -0500 )edit

@sjhalayka Thanks.I had the rows and columns around the wrong way. If I use clone() I now get an exception when using imshow()

 terminate called after throwing an instance of 'cv::Exception'what():  OpenCV(4.0.0) /home/ed/src/opencv-4.0.0/modules/highgui/src/window_gtk.cpp:146: error: (-215:Assertion failed) dst.data == widget->original_image->data.ptr in function 'cvImageWidgetSetImage'

From what I've read the depth values for kinect v1 are between 0-2047. The documentation for imshow() mentioned that if you use AUTOSIZE when you create the window it will be displayed @ original size. Do I need to do any conversion between CV16UC1 to CV8UC1 before calling imshow()?

Ed95 gravatar imageEd95 ( 2018-12-12 02:13:58 -0500 )edit

@berak. Thanks, that was the first thing I saw too and that's a great ptr explanation!

Ed95 gravatar imageEd95 ( 2018-12-12 02:18:30 -0500 )edit
1

@Ed95 the last one looks like a recent bug. (imshow can only show 8bit data). it was fixed in the meantime, but you need to update your opencv libs to latest master & rebuild.

in the meantime, convert your depth image manually to 8U before using imshow:

depthimg.convertTo(depthimg, CV_8U, 1.0/255);
imshow("depth", depthimg);
berak gravatar imageberak ( 2018-12-12 02:59:54 -0500 )edit

@berak. Thanks that was perfect. Converting to CV_8U fixed the exception.Will update libs. The window isn't appearing or being updated though. The code I have for displaying the frames is something like :

   cv::namedWindow("Depth", WINDOW_AUTOSIZE);  //in the main
   depthimg.convertTo(depthimg, CV_8U, 1.0/255);       // in the function
   imshow("Depth", depthimg);                                //in the function

No doubt something simple. Will look into it.

Ed95 gravatar imageEd95 ( 2018-12-12 03:58:07 -0500 )edit
berak gravatar imageberak ( 2018-12-12 04:00:09 -0500 )edit

I read that you need to use waitKey() with a value greater than 0, and the depth image resolution for v1 is only 320x240. Implementing waitkey() I now get a black screen. If I play with the buffer sizes I can get a low quality image/mirrored image, with some strange overlay when the kinect is moved/rotated.Think now it's an issue with buffer sizes and scaling issues.

static std::vector<uint16_t> kdepth( 640 * 480 * 4); 
static unsigned int dBufSize = 640 * 480 * 4;
int dWidth = 320;
int dHeight = 240;
Mat mDepth = Mat(dHeight, dWidth, CV_16UC1,&depth[0]).clone();

mDepth.convertTo(mDepth,CV_8U, 1.0/255);
imshow("Depth",mDepth);
waitKey(1);

kinect image of sd card adpater

Ed95 gravatar imageEd95 ( 2018-12-12 20:47:00 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2018-12-16 20:13:06 -0500

Ed95 gravatar image

Managed to produce a depth image using the kinect v1 camera with openCV 4. The issues were :

  1. Wrong pointer assignment creating the cv::Mat data structure (as pointed out by sjhalayka and berak)

  2. Bug in the openCV library with imshow() when using CV_16UC1 (as pointed out by berak). An upgrade from 4.0.0 to dev libs fixed this (or you can use convertTo()). Also needed to use waitkey() with imshow().

  3. Correcting the depth callback function in the sample code. The original code (cppview.cpp) created an overlaid of images when using imshow(). Replacing the code fixed the problem, The depth data size should be 16 bits (the rgb data is only 8 bits per channel).

    // access kinect depth buffer

    void DepthCallback (void* _depth, uint32_t timestamp) {
    
           Mtx::ScopedLock lock(m_depth_mutex);
    
            m_buffer_depth.clear();
    
            uint16_t* call_depth = static_cast<uint16_t*>(_depth);
    
            for (size_t i = 0; i < 640*480 ; i++) {
                m_buffer_depth.push_back(call_depth[i]);
            }
    
           m_new_depth_frame = true;
    
      }
    
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-12-11 15:18:08 -0500

Seen: 286 times

Last updated: Dec 16 '18