I've never actually done this, but here's what I would do. In order to account for distortion you must find the intrinsic matrix of your camera by performing camera calibration using a known pattern and then apply the transform described by that matrix.
For normal ~180 degree cameras I usually use a checkerboard pattern. You need about 5 image of the checkerboard in various poses. Try to cover the field of view, capture all the corners, and make sure that the various poses aren't just translations, i.e. rotate the checkerboard about each axis. With these images in hand you can calculate the distortion matrix and use it to undistort the images.
OpenCV provides an omnidirectional camera model that I think is useful for your purposes.