I think boardSize is incorrect in your code. boardSize should be equal to a number of internal corners, so in your case it should be cv::Size boardSize(5,4).