Problem with Intel Realsense SDK 2.0 and cv::cvtColor

edit

Hello

I'm not sure if the problem comes from the Intel Realsense library or from OpenCV: I have a simple program that just gets a RGB image from the Intel Realsense D435, converts the image to BGR with cvtColor and shows it with imshow.

This works until I activate the depth camera and the RGB camera of the Intel Realsense D435 at the same time. Then I get some frames where it seems like the conversion of the RGB image was not done by OpenCV. So red appears blue and vice versa. This happens in irregular intervals.

Here is my code:

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <opencv2/opencv.hpp>   // Include OpenCV API

int main(int argc, char * argv[]) try
{
    // Declare RealSense pipeline, encapsulating the actual device and sensors
    rs2::pipeline pipe;

    rs2::config cfg;

    //depth stream config
    cfg.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, 30);

    //colour stream config
    cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_RGB8, 30);

    // Start streaming
    pipe.start(cfg);

    const auto window_name = "Display Image";
    cv::namedWindow(window_name, cv::WINDOW_AUTOSIZE);

    while (cv::waitKey(1) < 0 && cvGetWindowHandle(window_name))
    {
        rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera
        rs2::frame colour = data.get_color_frame();

        // Query frame size (width and height)
        const int w = colour.as<rs2::video_frame>().get_width();
        const int h = colour.as<rs2::video_frame>().get_height();

        // Create OpenCV matrix of size (w,h) from the colorized depth data
        cv::Mat image(cv::Size(w, h), CV_8UC3, (void*)colour.get_data(), cv::Mat::AUTO_STEP);
        //convert from RGB to BGR
        cv::cvtColor(image, image, cv::COLOR_RGB2BGR);

        // Update the window with new data
        imshow(window_name, image);
    }

    return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}
catch (const std::exception& e)
{
    std::cerr << e.what() << std::endl;
    return EXIT_FAILURE;
}

I'm using the newest OpenCV version 3.4.3 and Intel Realsense SDK 2.0 version 2.16.0.