Unwarp segment of 360 degree fisheye lens

edit

I have a 360 degree image taken from a fisheye lens and I would like to unwarp a segment of it. i.e. select a point in the image and then unwarp a segment surrounding that point - like a digital pan-tilt-zoom within the image.

I can successfully use the C++ version of opencv to create a panoramic image from it by building a mapping array and using remap like so:

int warpedW = src.size().width;
int warpedH = src.size().height;

int unwarpedW = 800;
int unwarpedH = 180;

// Angle to start unwarping from
float angularOffset = 0.0f;

// Centre point
float warpedCx = warpedW / 2.0f;
float warpedCy = warpedH / 2.0f;

/// Create map_x and map_y with the same size as src:
map_x.create( unwarpedH, unwarpedW, CV_32FC1 );
map_y.create( unwarpedH, unwarpedW, CV_32FC1 );

float circFactor = 0.0f - TWO_PI / (float) unwarpedW;

for ( int dsty = 0; dsty < unwarpedH; ++dsty )
{
    float y = ( (float) dsty / (float) unwarpedH );
    float yfrac = MIN( 1.0f, MAX( 0.0f, y ) ); 

    float radius = yfrac * warpedCy;

    for (int dstx = 0; dstx < unwarpedW; ++dstx )
    {
        float angle = ( (float) dstx * circFactor ) + ( DEG_TO_RAD * angularOffset );

        // map the source pixel to the destination pixel
        map_x.at< float >( dsty, dstx ) = warpedCx + radius * cosf( angle );
        map_y.at< float >( dsty, dstx ) = warpedCy + radius * sinf( angle );
    }
}

dst.create( unwarpedH, unwarpedW, src.type() );

remap( src, dst, map_x, map_y, CV_INTER_LINEAR );

I think I should be able to create mapping matrices and do a similar set of steps but I am a bit lost on the maths on how to do this.

Is there perhaps an easier way of doing this in opencv? I found detail::FisheyeWarper and detail::SphericalWarper and thought perhaps their warpBackward() method might work but my experimentation has so far been unsuccessful in getting these to do anything at all without asserting.