How can I compare an ideal projection with a real projection?

edit

Well, a few ways to do it. I think the structured light module might actually be the easiest. That will help you determine any deformations with the surface, and maybe projector distortion.

The other way would be to calibrate a camera, use the projector to display a chessboard in different positions. Undistort the image using the camera calibration, then warp the corners of the projection into an appropriately sized frame. Then treat that as the image from a theoretical camera at the projector location. I'm not sure whether you would need to reverse the distortion into the projector or not, but it shouldn't be hard to try.

EDIT:

So, thinking about this, there are actually three separate problems.

1. The Surface is not flat. - Use Structured Light
2. The Projector lens has distortion - Use Camera Calibration in reverse.
3. The Projector is not projecting square - Use Homography

You can probably safely ignore the first unless you're trying to project on a round building or something.

For the second the procedure is as follows:

• Project a chessboard on the wall with a calibrated camera viewing it.
• Undistort captured images according to camera calibration.
• Find corners of projected image and warp into a (X by Y) projector resolution image so the corners are the corners of the image.
• Use that to run a normal camera calibration

For the third, it's pretty simple.

• Pick the shape the projection should look like in aspect ratio. (IE: Find the corners of the projection screen)
• Display a totally white image and find the corners.
• Find the largest rectangle of screen aspect ratio that fits within the white. OR within the screen, if the projection is bigger than the display screen.

EDIT #2:

How to adjust the shape of the projection to make it square.

I'm assuming you have moved the projector into the best position you can manage, and already adjusted the keystone to get it as close as possible to correct. But now you need to project some pattern in a uniform manner but the image isn't uniform.

The first step is to get your calibrated camera (since every little bit helps) and take a picture. I'm going to fake the photos in GIMP, it's quicker and you'll get the idea.

Then you need to get your tape measure and measure the projection screen. You need to do this to be sure of the aspect ratio so you can warp it as if you were facing it directly on. Measure it in the X and Y directions. Take your undistorted image and pick the corners of your screen (or surface) and warp them to a Mat of the same aspect ratio of the screen, which I'll call ScreenMat. It should be a larger resolution than your projection.

(See that? That's why you calibrate your camera and undistort first. Your lines should be straight, not bent, I think. If they aren't, you will need do a Camera ...
(more)