Real-time Stitching from camera not performing well [closed]

asked 2019-06-26 06:20:28 -0500

danzix gravatar image

Hello everyone! I am trying to make a real-time stitching function from camera in Unity. By learning some reference, I have been done a simple real-time stitching from camera and this is my current work. However, we can see there are some problem in the video. First, there is shaking every frame from the stitched video; second is the stitched video color is different from the left-hand side; and finally, there is a conspicuous line from the stitched video. Therefore, I want to know how to fix the problem and thanks everyone who seeing my question. Here is my code of the project, I have been using a Plugin called OpenCVforUnity.

    void OnStitch1()
{
    pr2 = Stitching(second_mat_gray, first_mat_gray);

    Utils.matToTexture2D(pr2, _Texture1);
    output.texture = _Texture1;
    output.GetComponent<AspectRatioFitter>().aspectRatio = (float)_Texture1.width / (float)_Texture1.height;
}

Mat Stitching(Mat first_mat_temp, Mat second_mat_temp)
{
    detector.detect(first_mat_temp, keypointsSrc);
    extractor.compute(first_mat_temp, keypointsSrc, descriptorsSrc);

    detector.detect(second_mat_temp, keypointsScene);
    extractor.compute(second_mat_temp, keypointsScene, descriptorsScene);

    matcher.clear();
    matcher.match(descriptorsSrc, descriptorsScene, matches);

    matchesList = matches.toList();

    for (int i = 0; i < descriptorsSrc.rows(); i++)
    {
        dist = (double)matchesList[i].distance;
        if (dist < min_dist)
            min_dist = dist;
        if (dist > max_dist)
            max_dist = dist;
    }

    good_matches.Clear();
    for (int i = 0; i < matchesList.Count; i++)
    {
        if (matchesList[i].distance < 10 * min_dist)
            good_matches.Add(matchesList[i]);
    }
    GM = good_matches.Count;

    keypoints_objectList = keypointsSrc.toList();
    keypoints_sceneList = keypointsScene.toList();
    keypointsSrc.release();
    keypointsScene.release();

    for (int i = 0; i < good_matches.Count; i++)
    {
        objList.Add(keypoints_objectList[good_matches[i].queryIdx].pt);
        sceneList.Add(keypoints_sceneList[good_matches[i].trainIdx].pt);
    }

    MatOfPoint2f obj = new MatOfPoint2f();
    MatOfPoint2f scene = new MatOfPoint2f();

    obj.fromList(objList);
    scene.fromList(sceneList);
    H = Calib3d.findHomography(obj, scene, Calib3d.RANSAC, 10);
    warp_mat = second_mat.clone();

    if (GM - gmCount > 10)
    {
        gmCount = good_matches.Count;
        tempH = H.clone();
        Imgproc.warpPerspective(second_mat, warp_mat, H, ims);
    }
    else
    {
        Imgproc.warpPerspective(second_mat, warp_mat, tempH, ims);
    }
    if (gmCount > 400 || GM < 100)
        gmCount = GM;


    half = new Mat(warp_mat, _rect);
    first_mat.copyTo(half);
    return warp_mat;
}
edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by sturkmen
close date 2020-09-18 06:02:20.514633

Comments

sturkmen gravatar imagesturkmen ( 2019-06-26 07:17:46 -0500 )edit

Thanks for your reply, This link is really useful. I find video stabilizing can be solved the shaking problem. Also, I think find Seam and then using addWeighted() is the key to clear the conspicuous line from the stitched video, but I have no idea how to find the seam position from the warp part, do you know how to do that?

danzix gravatar imagedanzix ( 2019-06-26 10:14:03 -0500 )edit