reprojectImageTo3D gives weird results

edit

EDIT: So it turns out that the function was working as intended. I just had to change the clouds to the 'original scene' colour to realize that I was at a very bad viewing angle.. Im an idiot.. Results below

So I guess my next question which i'll post in a different thread would be how to initialize the camera position and viewing angle such that the 3D scene fills the screen and is viewed such that it looks similar to the image on the 1st picture.

Also berak, what did you mean by x,y ? If you meant pixel col and row then I would get a 'flat' 3D scene like below. I am not able to get a nice cube like you mentioned.

    Mat disparity = imread("2011_09_26_drive_0095//0000000099_left_disparity.png", CV_LOAD_IMAGE_GRAYSCALE);
    Mat image = imread("2011_09_26_drive_0095//image//0000000099.png", CV_LOAD_IMAGE_GRAYSCALE);

    namedWindow("Disparity Map", CV_WINDOW_AUTOSIZE); imshow("Disparity Map", disparity);
    namedWindow("Image", CV_WINDOW_AUTOSIZE); imshow("Image", image);

    disparity.convertTo(disparity, CV_32F);
    disparity = disparity / 256.0;

    //----------------------------------------------------------------------
    // Reconstruct
    //----------------------------------------------------------------------

    cv::Mat XYZ(disparity.size(), CV_32FC3);
    reprojectImageTo3D(disparity, XYZ, Q);

    //----------------------------------------------------------------------
    // Visualize
    //----------------------------------------------------------------------

    /// Create a window
    viz::Viz3d myWindow("Coordinate Frame");

    while (!myWindow.wasStopped())
    {
        /// Create a cloud widget
        viz::WCloud cw(XYZ, viz::Color::red());

        /// Display it in a window
        myWindow.showWidget("CloudWidget1", cw);

        myWindow.spinOnce(1, true);
    }