3d reconstruction Autocalibrate? Stereocalibrate?

edit

Hello

I am interested in 3D Reconstruction.

Given...

a) camera matrix and distortion coefficients

b) matched image points

c) triangulated 3d points from matched image points

d) 3x4 Projection matrices from solvePnp for each view

In order to do dense reconstruction, we need to get K1 and K2 for stereoRectify() with matching pairs of views. An autocalibration step is advised, but the literature is a bit sparse on how to do this.

I have attempted to use stereoCalibrate with the objects points as my points cloud, and the corresponding images points. I have essentially replaced the chessboard with the triangulated 3D points.

std::pair<std::vector<cv::Point2f>,std::vector<cv::Point2f>> imagePointsPairs = sfm.PointsForViews(i,j);    

std::vector<cv::Point2f> imagePoints1 = imagePointsPair.first;  
std::vector<cv::Point2f> imagePoints2 = imagePointsPair.second;
patternPts1Vec.push_back(imagePoints1);
patternPts2Vec.push_back(imagePoints2);

std::vector<cv::Point3f> objectPoints=sfm.CloudPointsForViews(2,6);
std::vector<std::vector<cv::Point3f>> objectPtsVec;
objectPtsVec.push_back(objectPoints);
cv::Mat rMat,tMat,F,E;  
cv::Mat cameraMatrix1 = cv::Mat::eye(3, 3, CV_32F);
cv::Mat cameraMatrix2 = cv::Mat::eye(3, 3, CV_32F);     
double rms = cv::stereoCalibrate(cv::Mat(objectPointsVec), cv::Mat(imagePoints1Vec),cv::Mat(imagePoints2Vec),
                    cameraMatrix1,sfm.distortion_coeff,
                    cameraMatrix2,sfm.distortion_coeff,
                    imgs[2].size(), rMat, tMat, E, F,
                    cv::TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, 1e-5),
                    CV_CALIB_ZERO_TANGENT_DIST +
                    CV_CALIB_SAME_FOCAL_LENGTH);

This raises an assertion

OpenCV Error: Assertion failed (ni >= 0) in cv::collectCalibrationData