Is my Disparity Map wrong?

edit

numDisparities=24 <-- means, you get 24 slices, so yes, that's expected.

Oh.. yes you are right.. i didn't think from this point of view..
But why the slides of my point cloud have that strange distance?

btw, above is without Q, right ?

It is... i made:

Q <<   1, 0, 0, -cx,
       0, 1, 0, -cy,
       0, 0, 0,   f,
       0, 0,-1,   0;


Mat_<Vec3f> XYZ(image_disparity.rows, image_disparity.cols);   // Output point cloud
Mat_<float> vec_tmp(4, 1);

for (int y = 0; y < image_disparity.rows; ++y) {
    for (int x = 0; x < image_disparity.cols; ++x) {
        vec_tmp(0) = x;
        vec_tmp(1) = y; 
        vec_tmp(2) = image_disparity.at<float>(y, x);
        vec_tmp(3) = 1;

        vec_tmp = Q * vec_tmp;

        Vec3f &point = XYZ.at<Vec3f>(y, x);
        point[0] = vec_tmp(0);
        point[1] = vec_tmp(1);
        point[2] = vec_tmp(2)/vec_tmp(3);
    }
}

^^ ah, cool. now at least, we've got something, we can send "the next one" to ;)

but honestly, 3d from disparity is hard , even with a proper calibrated stereo rig (which you don't have).

and yea, your disparity looks like sh..t (it's not a problem with the viewer, or with your projection formula)

keep on experimenting, but rather try to simulate, what a stereo cam (or 2 human eyes) would see.

e.g. i think, your images are "too far apart", and the direction is not "parallel" enough (don't start me on the cars ...)

for comparison, here's a "nice" disparity image

may i ask, what your "use-case" is ?

for certain situations (small things, indoor) there are alternatives to block-matching (photogrammetric stereo, structured light, etc)

it should be used for an application similar to Google Streetview... so outdoor usage