I'm trying out fisheye calibration with samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp. I've adapted the example VID5.xml file, then calibrated without fisheye, and get a 1.36 pixel reprojection error. Next I added <Calibrate_UseFisheyeModel>1</Calibrate_UseFisheyeModel>, and get an exception:
Re-projection error reported by calibrateCamera: 282.075
Calibration succeeded. avg re projection error = 282.075
OpenCV Error: Assertion failed (rvecs[i].rows == 3 && rvecs[i].cols == 1) in saveCameraParams, file /home/lwalter/other2/opencv_latest/samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp, line 611
terminate called after throwing an instance of 'cv::Exception'
what(): /home/lwalter/other2/opencv_latest/samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp:611: error: (-215) rvecs[i].rows == 3 && rvecs[i].cols == 1 in function saveCameraParams
Aborted (core dumped)
This is using the latest source, I could try going back to a release though I'll have to track down when the useFisheye logic was added to camera_calibration.cpp. I'll run this through a debugger as well.