How to get bird view image which is synthized four side images?(get AVM: around view monitoring)

asked 2020-12-15 19:32:24 -0500

gunny gravatar image

updated 2020-12-16 10:49:12 -0500

berak gravatar image

Hi I'm making avm(around view monitoring) image by getting four side images from four fisheye cameras. This process has three parts:

  1. calibrate fisheye image/video/streaming
  2. get top view(bird view) image
  3. get avm image by synthesizing four side images

I has completed 1 and 2 parts, but I'm in trouble doing third part. I researched it and I found that it is may related to 'stitching' or 'numpy array'. But I don't know how to do it although I read documentation.

+add) presentresult image C:\fakepath\AVM_test.png

code: (brought from github Ahid-Naif/Around-View-Monitoring-AVM)

import cv2
import numpy as np
import imutils
from Camera.Undistortion import UndistortFisheye
from Camera.PerspectiveTransformation import EagleView
# from Camera.Stitcher import stitchTwoImages
import time

class avm:
    def __init__(self):
        self.__leftCamera = UndistortFisheye("left_Camera")
        self.__rightCamera = UndistortFisheye("right_Camera")

        self.__leftEagle = EagleView()
        self.__rightEagle = EagleView()
        # self.__frontEagle.setDimensions((149, 195), (439, 207), (528, 380), (37, 374))
        # self.__backEagle.setDimensions((164, 229), (469, 229), (588, 430), (45, 435))

        #reset left/right setDimensions
        self.__leftEagle.setDimensions((186, 195), (484, 207), (588, 402), (97, 363))
        self.__rightEagle.setDimensions((171, 240), (469, 240), (603, 452), (52, 441))
        # self.__leftEagle.setDimensions((186, 195), (484, 207), (588, 402), (97, 363))
        # self.__rightEagle.setDimensions((171, 240), (469, 240), (603, 452), (52, 441))

        self.__middleView = None
        self.__counter = 0

        # self.stitcher = stitchTwoImages("Bottom2Upper")
        # self.upper = None
        # self.bottom = None

    def runAVM(self, leftFrame, rightFrame):
        leftView = self.__leftCamera.undistort(leftFrame)
        topDown_left = self.__leftEagle.transfrom(leftView)
        rightView = self.__rightCamera.undistort(rightFrame)
        topDown_right = self.__rightEagle.transfrom(rightView)
        # topDown_Back = cv2.flip(topDown_Back, 1) #flip left/right

        topDown_left , topDown_right = self.__reScale(topDown_left, topDown_right)
        # stitchingResult = self.__startStitching(topDown_Front)
        middleView = self.__getMiddleView(topDown_left)
        birdView = np.hstack((topDown_left, middleView, topDown_right))
        return birdView

    def __reScale(self, topDown_left, topDown_right):
        width_leftView = topDown_left.shape[1]
        width_rightView = topDown_right.shape[1]
        height_leftView = topDown_left.shape[0]
        height_rightView = topDown_right.shape[0]
        if height_leftView > height_rightView:
            newHeight = height_rightView
            ratio = height_rightView/height_leftView
            newWidth = int(ratio * width_leftView)
            topDown_left = cv2.resize(topDown_left, (newWidth, newHeight))
        else:
            newHeight = height_leftView
            ratio = height_leftView/height_rightView
            newWidth = int(ratio * width_rightView)
            topDown_right = cv2.resize(topDown_right, (newWidth, newHeight))

        return topDown_left, topDown_right

    def __getMiddleView(self, topDown_left):
        # the length of the image represents the distance in front or back of the car
        width_leftView = topDown_left.shape[1]
        if self.__middleView is None:
            realWidth_leftView = 13 # unit is cm
            realWidth_MiddleView = 29.5 # unit is cm
            ratio = int(width_leftView/realWidth_leftView)
            width_MiddleView = int(realWidth_MiddleView * ratio)
            height_MiddleView = int(topDown_left.shape[0])  
            self.__middleView = np.zeros((height_MiddleView, width_MiddleView//2, 3), np.uint8)
            # print(ratio)
        # else:
        #     #  self.__middleView[0:stitchingResult.shape[0], :]

        return self.__middleView

    # def __startStitching(self, accView):
    #     if self.bottom is None:
    #         self.bottom = accView
    #         return None
    #     else:
    #         # time.sleep(0.5)
    #         self.upper = accView
    #         self.bottom = self.stitcher.stitch(self.bottom, self.upper)
    #         cv2.imshow("Result", self.bottom)
    #         height = accView.shape[0]
    #         return self.bottom[height:self.bottom.shape[0], :]

If you know anythings about it, please comment and share your thoughts. Thank you for your reading. (I'm using opencv, python, raspberrypi 4B)

edit retag flag offensive close merge delete

Comments

please be more explicit, about how the outcome should look like. do you want an equirectangular projection ?

how much do your images overlap ? are the cameras fixed ? showing a set of example images might be useful here

isit something like this ?

berak gravatar imageberak ( 2020-12-16 03:16:04 -0500 )edit

OH, first thank you for your comments and interests about my problem:)

yes, those images perfectly show what I mentioned. 4 corner overlaps exist(topleft, topright, bottomleft, bottomright).4 cameras are fixed. I don't know how to synthesize four images, and present result image is below:

result image

I used np.hstack for stitch three view(leftview, middleview, rightview)

Thank you for reading:)

gunny gravatar imagegunny ( 2020-12-16 08:07:19 -0500 )edit

i'm quite sure you need some sort of remapping for this.

unfortunately, i'm afraid, the exact projection formulas are some kind of "trade secret" here ;(

berak gravatar imageberak ( 2020-12-16 10:46:37 -0500 )edit

oh, LoL That's why I could not find open source in effort of 2 week... (BTW I used "remapping" part by undistorting the fisheye images.)

Although I don't find exact solution about it, thank you for your attention!

gunny gravatar imagegunny ( 2020-12-16 21:44:06 -0500 )edit