ricor's profile - activity

Ok it will never be real time and I don't think anybody was necessarily expecting it to be but it certainly isn't out of the realms of possibility to speed this up substantially.

Hi Nightlife,

I've been looking at the stitching_detailed.cpp example which is heavily based on the stitching class. So far I've noticed several things that could help with speed that may be relevant to you whether or not you are using that example:

1. The feature detection for each image is run within a for loop in the example. This could be parellelised to run on multiple cores as getting the features in each image is independent.

2. There are appear to be some GPU capabilities for SURF and I think other places within Stitching.

3. What resolution are you using for the image registration stage? Can this be lowered without affecting performance?

4. What resolution are you using for the composition stage? Can this be lowered and still give you adequate panorama? I've found that this can affect performance a lot.

Can you tell us where your bottleneck is (feature detection, pairwise matching, estimating homography, composition)? How many images are you trying to stitch etc?

When examining the stiching module it appears to only be setup for a rotating camera. In theory can it be modified easily for a rotating and translating camera?

I think it is in the bundle adjustment that this would need to be done because in the Brown and Lowe paper which describes the stitching module it states:

The new image is initialised with the same rotation and focal length as the image to which it best matches. Then the parameters are updated using Levenberg-Marquardt

Is this where effort should be made to make the camera rotate and translate (this is all new to me)?

Within the bundle adjustment can I just initialise the next image with the last images rotation combined with a best estimate on translation? Somehow that doesn't seem right to me as I've read about cost functions etc. Do I need to do something actually inside the bundle adjustment code?

I'm quite lost here and googling hasn't turned anything up. I'm very new to this though so perhaps I'm searching with the wrong terminology. If anybody has any suggestions at all even it is just some term to try googling it would really help.

Thanks

I've tried using the stitching_detailed example and while it works for a small set of images for more numerous cases it doesn't (e.g. >15). I find that in the camera estimation matrix log messages it is returning nan's. This of course means no panorama. I'm quite new to image processing so I'm struggling to understand what is going on here but I think it is to do with estimation of the focal length from the camera matrix.

I've looked for help in the docs but only found the opencvdoc API pages. I've therefore dug into the code in the stitching module and had a look at the offending bit in motion_estimators.cpp (code at the bottom).

However, can anyone help me in understanding with what is going on in that bit of code (below). In particular why is there a commented out bit (#if 0 to #endif) and what advantage would that give me? Also would this code be expected to be numerically unstable - probably most likely to be unstable in the call that reaches focalsFromHomography()?

Finally does anybody know of any explanatory docs to explain focalsFromHomography? It doesn't look like a factorisation of the camera matrix to me (which it possibly is in a different function where it starts talking about cholesky decomp). However, I'm very new to image processing so don't really know what I'm talking about here.

Many thanks for any tips on this

void HomographyBasedEstimator::estimate(const vector<ImageFeatures> &features, const vector<MatchesInfo> &pairwise_matches,
                                        vector<CameraParams> &cameras)
{
    LOGLN("Estimating rotations...");
#if ENABLE_LOG
    int64 t = getTickCount();
#endif

    const int num_images = static_cast<int>(features.size());

#if 0
    // Robustly estimate focal length from rotating cameras
    vector<Mat> Hs;
    for (int iter = 0; iter < 100; ++iter)
    {
        int len = 2 + rand()%(pairwise_matches.size() - 1);
        vector<int> subset;
        selectRandomSubset(len, pairwise_matches.size(), subset);
        Hs.clear();
        for (size_t i = 0; i < subset.size(); ++i)
            if (!pairwise_matches[subset[i]].H.empty())
                Hs.push_back(pairwise_matches[subset[i]].H);
        Mat_<double> K;
        if (Hs.size() >= 2)
        {
            if (calibrateRotatingCamera(Hs, K))
                cin.get();
        }
    }
#endif

    if (!is_focals_estimated_)
    {
        // Estimate focal length and set it for all cameras
        vector<double> focals;
        estimateFocal(features, pairwise_matches, focals);
        cameras.assign(num_images, CameraParams());
        for (int i = 0; i < num_images; ++i)
            cameras[i].focal = focals[i];
    }
    else
    {
        for (int i = 0; i < num_images; ++i)
        {
            cameras[i].ppx -= 0.5 * features[i].img_size.width;
            cameras[i].ppy -= 0.5 * features[i].img_size.height;
        }
    }

    // Restore global motion
    Graph span_tree;
    vector<int> span_tree_centers;
    findMaxSpanningTree(num_images, pairwise_matches, span_tree, span_tree_centers);
    span_tree.walkBreadthFirst(span_tree_centers[0], CalcRotation(num_images, pairwise_matches, cameras));

    // As calculations were performed under assumption that p.p. is in image center
    for (int i = 0; i < num_images; ++i)
    {
        cameras[i].ppx += 0.5 * features[i].img_size.width;
        cameras[i].ppy += 0.5 * features[i].img_size.height;
    }

    LOGLN("Estimating rotations, time: " << ((getTickCount() - t) / getTickFrequency()) << " sec");
}