Goal: Capture images of a planar surface using two fully calibrated cameras, and process the images to generate a single undistorted, scaled, rectilinear* image.
*I'm probably misusing the word rectilinear. What I mean is that the world plane being imaged is mapped to the processed image so that world coordinate [Wx,Wy,0] maps to image pixel [sWx,sWy] in the processed image. Uniform scale, angles are preserved, etc. A similarity transform from world plane to image plane, I suppose.
Good news: I can do this successfully with the following steps: 1. Capture images. 2. Undistort and scale images using cvInitUndistortRectifyMap() + remap() 3. compute perspective transform between undistorted image coords to output image coords: cv::getPerspectiveTransform() 4. apply perspective warp with cv::warpPerspective()
Works great.
Bad news: I can't figure out how to apply the perspective transform as part of the cvInitUndistortRectifyMap()
I would like to cut out the final step, and include it as part of the remap() call. Initially I thought I could use the R parameter to specify a perspective transform (or the newCameraMatrix parameter), but I haven't been able to get this to work.
In looking at the code for initUndistortRectifyMap() it seems that the R parameter needs to apply the transform in normalized camera coordinates. I say this because large translations that I would get from my perspective warp (say 500 pixels) are completely wrong, and instead I need translations in a much smaller range to make sense (say, less than 0.5). I tried dividing the translation component by the focal length of my camera matrix, which "helped", but didn't result in an accurate image.
Questions: Can I use the initUndistortRectifyMap() function to achieve what I am trying to do? If so, should I be trying to come up with the right R parameter, or trying to come up with the right newCameraMatrix parameter, or...are they interchangeable for my purposes?
I tried to deconstruct the code for initUndistort... but it was difficult to follow fully. In particular I became concerned that the R / newCameraMatrix parameters would affect how undistortion was applied (which seems unlikely, so I concluded that I was not following the code correctly)
It sure seems like I should be able to use the R matrix to rectify both images to a common plane (which requires a full 2D homography (including translation) in my case), but I'm just not quite able to get it to work.
Help?