Perspective transform and lens undistortion in one step?

edit

I don't think I can do it with a chain of matrices as the my distortions are quite strong and cannot be represented by a simple linear transformation in polar coordinates. But I might be wrong.

The problem is I do not have any parameters of the camera. I could theoretically calibrate it using parallel lines and vanishing points (not really sure how to do it), undistort the image and then find the perspective transform...

I was rather looking for some transformation (and interpolation) between two sets of 3D points. One set consist of points in the image X,Y and Z (Z=0 for street level, curb will be, let's say, Z=0,1 etc) and the other set consists of real-world X,Y,Z (Z should be the same) coordinates of these points. Since I am not able to cover the entire image with dense points...

Since I am not able to cover the entire image with dense points, I would like to be able to use the screen coords (with extra assumed Z coordinate) to estimate the real-world coordinates and be able to interpolate between the actually measured points. How do I calculate that?

Ok, so I suppose you have a static camera you don't know the parameters. Then you should build a displacement map. It's basically a discrete function f(p_image)=p_world.

You take as many points from the image as you can and define the real world positions. Try to stay in 2D (x,y->x',y'). Then, get the whole displacement image by interpolating the pixels defined in the previous step. As the lines in the image are curved, you should use a spline-based interpolation (like bicubic).

To get the world Z coordiantes, you could simply draw a mask (street=0, curb=1, grass=2...)

OK then, is there a tool that would do this interpolation and generate a 2D displacement image with, say, 1 pixel density? I'm sure there is. interp3 in Matlab? Or interp2 separately for X and Y?

In this case you'll have a sparse matrix (<100 elements with random position for the whole image). To interpolate it, you'll have to use something like ScatteredInterpolant or inpaint_nans in Matlab.

Thank you all for suggestions. I will give it probably a try when I have some free time. I did contact some people using this method, and it turned out that they first do the calibration (Tsai model) and than undistortion with perspective transformation. Unfortunately, I am afraid that Tsai model is not good enough with strong lens distortions... I would probably need a more precise method of esimating camera parameters from a single view and matched 2D points. Anyone knows one?