Detect the bounds of a Passport page with OpenCV

asked 2016-10-22 08:49:13 -0500

iamMehedi gravatar image

updated 2016-10-30 00:41:58 -0500

I am trying to develop a scanner that can scan a page of a Passport with the camera.

So from a Passport page like this: image description

I'd like to crop out the marked part.

I have written code for edge detection using OpenCV which finds the contours and then approximates the largest quadrilateral. Finally it does a 4 point perspective transformation to get a top view of the image. The edge detection code look like this:

public static List<MatOfPoint> findContours(Mat src){
Mat img = src.clone();
src.release();
//find contours
double ratio = getScaleRatio(img.size());
int width = (int) (img.size().width / ratio);
int height = (int) (img.size().height / ratio);
Size newSize = new Size(width, height);
Mat resizedImg = new Mat(newSize, CvType.CV_8UC4);
Imgproc.resize(img, resizedImg, newSize);

Imgproc.medianBlur(resizedImg, resizedImg, 5);

Mat cannedImg = new Mat(newSize, CvType.CV_8UC1);
Imgproc.Canny(resizedImg, cannedImg, 70, 200, 3, true);
resizedImg.release();

Imgproc.threshold(cannedImg, cannedImg, 200, 255, Imgproc.THRESH_OTSU);

Mat dilatedImg = new Mat(newSize, CvType.CV_8UC1);
Mat morph = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(3, 3));
Imgproc.dilate(cannedImg, dilatedImg, morph, new Point(-1, -1), 2, 1, new Scalar(1));
cannedImg.release();
morph.release();

ArrayList<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(dilatedImg, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
hierarchy.release();

Log.d(TAG, "contours found: " + contours.size());

Collections.sort(contours, new Comparator<MatOfPoint>() {
    @Override
    public int compare(MatOfPoint o1, MatOfPoint o2) {
        return Double.valueOf(Imgproc.contourArea(o2)).compareTo(Imgproc.contourArea(o1));
    }
});

return contours;
}

this code works for single page documents i.e ID cards, credit cards. Where the input is like this:

image description

But doesn't work for passports as the top edge is not as distinctive, for example: image description

The inputs will be taken from camera preview on Android. Any idea how can I detect the passport page? I am using OpenCV 3.1. Thanks in Advance!

EDIT: I am adding a few sample images.

1. Sample 1

2. Sampe 2

3. Sample 3

4. Sample 4

EDIT 2: I have tried drawing the top border myself so that a quadrilateral could be formed, but in that case the perspective transformation doesn't work as expected. Here's the complete code:

img = data.input.clone();
                    data.input.release();
                    //find contours
                    ratio = img.size().height/FIXED_HEIGHT;
                    width = (int) (img.size().width / ratio);
                    height = (int) (img.size().height / ratio);
                    newSize = new Size(width, height);
                    resizedImg = new Mat(newSize, CvType.CV_8UC4);
                    Imgproc.resize(img, resizedImg, newSize);
                    onNextStep(resizedImg);

                    Imgproc.medianBlur(resizedImg, resizedImg, 5);
                    onNextStep(resizedImg);

                    //Imgproc.line(resizedImg, new Point(0, 0), new Point(resizedImg.cols()-1, 0), new Scalar(0, 0, 0), 1);

                    cannedImg = new Mat(newSize, CvType.CV_8UC1);
                    Imgproc.Canny(resizedImg, cannedImg, 70, 200, 3, true);
                    resizedImg.release();
                    onNextStep(cannedImg);

                    Imgproc.threshold(cannedImg, cannedImg, 70, 255, Imgproc.THRESH_OTSU);
                    onNextStep(cannedImg);

                    dilatedImg = new Mat(newSize, CvType.CV_8UC1);
                    morph = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(3, 3));
                    Imgproc.dilate(cannedImg, dilatedImg, morph, new Point(-1, -1), 2, 1, new Scalar(1));
                    cannedImg.release();
                    morph.release();
                    onNextStep(dilatedImg);

                    contours = new ArrayList<>();
                    hierarchy = new Mat();
                    Imgproc.findContours(dilatedImg, contours, hierarchy, Imgproc ...
(more)
edit retag flag offensive close merge delete

Comments

What is the issue with the second example? There is only a blue border missing which is not an issue right?

StevenPuttemans gravatar imageStevenPuttemans ( 2016-10-24 08:13:17 -0500 )edit

@StevenPuttemans The 2nd example tries to demonstrate the problem I am facing while trying to scan an actual Passport with the device's camera. The algorithm for finding the largest contour doesn't work as it can't detect the edge on the top side, hence cannot find a quadrilateral.

iamMehedi gravatar imageiamMehedi ( 2016-10-26 01:29:00 -0500 )edit

Euhm why not simply add a small blue border yourself? o_O

StevenPuttemans gravatar imageStevenPuttemans ( 2016-10-26 02:48:02 -0500 )edit

@StevenPuttemans Tried that actually. Due to the document's & device's orientation the continuous end can be on any side so I am having a hard time determining where to draw that line. And drawing a line end to end messes up the perspective transform a bit.

iamMehedi gravatar imageiamMehedi ( 2016-10-27 08:35:26 -0500 )edit

Then please provide more challenging images where it gos wrong. In the last image, simply segmenting BLUE, will give you all 4 borders, then you can bump the top with 5 lines of blue and a border will be available.

StevenPuttemans gravatar imageStevenPuttemans ( 2016-10-28 04:44:44 -0500 )edit

@StevenPuttemans the goal is to detect the Passport and crop out the passport from the image and get a top view of it. As it'll be done with the device camera, background can be of any color (preferably darker than the passport). I'll share a few sample images shortly.

iamMehedi gravatar imageiamMehedi ( 2016-10-28 23:12:58 -0500 )edit

@StevenPuttemans added a few samples. Thanks!

iamMehedi gravatar imageiamMehedi ( 2016-10-29 00:00:43 -0500 )edit
1

I don't think it is legal to share real passport images. At least you could have edited the images to hide information...

LorenaGdL gravatar imageLorenaGdL ( 2016-10-29 04:52:23 -0500 )edit

@LorenaGdL all those passport images are collected from Google image search. I didn't think of the security issue as they're already out there on the web.

iamMehedi gravatar imageiamMehedi ( 2016-10-29 11:26:43 -0500 )edit
1

Redistributing can also be a crime... but It's definitely a socially irresponsible thing to do, that's potentially unsafe for the exposed people.

LorenaGdL gravatar imageLorenaGdL ( 2016-10-29 11:47:02 -0500 )edit