I found this on StackOverflow, which answers the question. The sum must be calculated inside the root. However the formula the answer provides is slightly incorrect, as the formula should be: \sqrt(\sum(diff(x-x0)²/n) https://stackoverflow.com/questions/23781089/opencv-calibratecamera-2-reprojection-error-and-custom-computed-one-not-agree?rq=1