horrible calibration results

edit

Hi,

I tried to integrate the calibration sample in my program but the results I'm getting are quite horrible. I think I've copied the essential parts correctly but the results are just wrong. For my webcam (which i believe has near 0 distortion) the results are decent but for my ip camera with 98 degree fov x the results are completely random (spherical distortion with holes in it sometimes)

As you can see that can't be right. I've read all the dovumentation i could find but can't find anything that could cause this. (I've tried to swap Cols/Rows and all the flags etc)

The code

void Worker::runCalibration(){

calibChessFound = findChessboardCorners( iMat, Size(calibCols,calibRows), calibPointBuf,
                               CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_FAST_CHECK | CV_CALIB_CB_NORMALIZE_IMAGE);
if(calibChessFound){
    flashTime = QTime::currentTime().msec();
    Mat iMatGray;
    cvtColor(iMat,iMatGray,CV_BGR2GRAY);
    cornerSubPix(iMatGray,calibPointBuf,Size(11,11),Size(-1,-1),TermCriteria(CV_TERMCRIT_EPS+CV_TERMCRIT_ITER,30,0.1));

    calibImagePoints.push_back(calibPointBuf);

    int flags=0;
    if(calibFAR){flags = flags|CV_CALIB_FIX_ASPECT_RATIO;}
    if(calibFPP){flags = flags|CV_CALIB_FIX_PRINCIPAL_POINT;}
    if(calibAZT){flags = flags|CV_CALIB_ZERO_TANGENT_DIST;}

    if(calibImagePoints.size() > 10){

        calibCamMat = Mat::eye(3, 3, CV_64F);
        if( calibFAR ){
            calibCamMat.at<double>(0,0) = iMat.cols/iMat.rows;
        }
        calibDistCoeffs = Mat::zeros(8, 1, CV_64F);

        vector<vector<Point3f> > objectPoints(1);
        calcBoardCornerPositions(Size(calibCols,calibRows), calibSize, objectPoints[0]);
        objectPoints.resize(calibImagePoints.size(),objectPoints[0]);

        double rms = calibrateCamera(objectPoints, calibImagePoints, iMat.size(), calibCamMat,
                        calibDistCoeffs, rvecs, tvecs, flags|CALIB_FIX_K4|CALIB_FIX_K5);
                        //*|CALIB_FIX_K3*/|CALIB_FIX_K4|CALIB_FIX_K5);
        emit error("RMS error of camera calibration is, rms = "+QString::number(rms),1);
        bool ok = checkRange(calibCamMat) && checkRange(calibDistCoeffs);
        if(!ok){
            emit error("INVALID VALUES @ Worker::runCalibration",1);
        }else{
            initUndistortRectifyMap(calibCamMat,calibDistCoeffs,Mat(),
                                    getOptimalNewCameraMatrix(calibCamMat,calibDistCoeffs,iMat.size(),1),
                                    iMat.size(),CV_16SC2,defLensMap1,defLensMap2);
        }
    }

}
}

Any ideas? I'm running out of the fast.

Thanks!