Detecting Documents On White Surface

asked 2016-06-13 17:52:59 -0600

I am trying to reproduce a real time document scanner like the one Scannable evernote has created. With some of the methods I have found and tweaked, I was able to create a reliable outline of a document, but against surfaces that are not glossy and the color must be darker than the document. I need help finding a way to capture the document on a glass desk or on a surface that looks white, but the human eye can still make a clear distinction that between the document and surface color. In addition, these methods can be very slow.

Method One: (Cannot Detect Documents On White Surface)

cv::Mat downScale, upScale, hsv, whiteScale, grayScale, threshold, smooth;
 cv::Mat grayContours;

 cv::pyrDown(image, downScale, cv::Size(image.cols/2, image.rows/2));
 cv::pyrUp(downScale, upScale, image.size());

      cv::cvtColor(upScale, hsv, COLOR_RGB2HSV);
      cv::inRange(hsv, cv::Scalar(0, 0, 0), cv::Scalar(0, 0, 255), whiteScale);
      cv::cvtColor(whiteScale, grayScale, CV_RGB2GRAY);
      cv::cvtColor(upScale, grayScale, CV_RGB2GRAY);

 //cv::medianBlur(grayScale, smooth, 3);
 //cv::GaussianBlur(grayScale, smooth, cv::Size(3, 3), 0, 0);
 cv::blur(grayScale, smooth, cv::Size(3,3));
 // 128, 255
 cv::threshold(smooth, threshold, 128, 255, THRESH_BINARY | THRESH_OTSU);

 cv::Canny(threshold, grayContours, 0, 50, 5);
 cv::dilate(grayContours, grayContours, cv::Mat(), cv::Point(-1,-1));

 vector<vector <cv::Point>> contours;
 findContours(grayContours, contours, RETR_LIST, CHAIN_APPROX_SIMPLE);

 vector<vector<cv::Point> > squares;
 vector<cv::Point> approx;
 for( size_t i = 0; i < contours.size(); i++ )
      approxPolyDP(Mat(contours[i]), approx, arcLength(Mat(contours[i]), true)*0.02, true);

      if( approx.size() == 4 &&
           fabs(contourArea(Mat(approx))) > 10000 &&
           isContourConvex(Mat(approx)) )
           double maxCosine = 0;

           for( int j = 2; j < 5; j++ )
                cv::Point pt1 = approx[j%4], pt2 = approx[j-2], pt0 = approx[j-1];
                double dx1 = pt1.x - pt0.x;
                double dy1 = pt1.y - pt0.y;
                double dx2 = pt2.x - pt0.x;
                double dy2 = pt2.y - pt0.y;

                double cosine = fabs((dx1*dx2 + dy1*dy2)/sqrt((dx1*dx1 + dy1*dy1)*(dx2*dx2 + dy2*dy2) + 1e-10));
                maxCosine = MAX(maxCosine, cosine);

           if( maxCosine < 0.3 ){
                detected = true;

      detected = false;

 for( size_t i = 0; i < squares.size(); i++ )
      const cv::Point* p = &squares[i][0];
      int n = (int)squares[i].size();
      cv::polylines(image, &p, &n, 1, true, Scalar(0,255,0), 3, CV_AA);

Method Two(Cannot Detect Document on White Surface):

 cv::RNG rng(12345);
 cv::Mat oResult, oToShow;
 cv::cvtColor(image, oToShow, CV_RGB2GRAY);

 //cv::threshold(image, oResult, 127, 255, 0);
 cv::threshold(oToShow, oResult, 128, 255, THRESH_BINARY | THRESH_OTSU);
 std::vector<std::vector<cv::Point>> contours;
 std::vector<cv::Vec4i> hierarchy;

 cv::findContours(oResult, contours, hierarchy, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);

 std::vector<std::vector<cv::Point> > oHull(contours.size());
 //vector<cv::Rect> boundRect( contours.size() );

 double largestArea = 0.0;
 if(contours.size() > 0)
 largestArea = cv::contourArea(contours[0]);
 int index = 0;
 for( size_t i = 0; i != contours.size(); i++)
      double area = cv::contourArea(contours[i]);
      if(area > largestArea && area > 50000){
           cv::convexHull(contours[i], oHull[i]);
           cv ...
edit retag flag offensive close merge delete


Can you add a sample image that is one you're having problems with? This is really not enough to go on.

Tetragramm gravatar imageTetragramm ( 2016-06-13 23:00:03 -0600 )edit

I can't provide you with an exact example, but these are the types of documents I am trying to capture. And it is against a cream like color surface table. So opencv is having a hard to finding a difference between the table and the document for many methods I tried. They are usually white with tables in them, with standard A4 printing paper size. I think I have an idea of what evernote has created. I believe that they analyze the image for a couple of seconds until it is sure that it has a highlighted document, and then it display the overlay rectangle on top of the document. Then it analyzes it again in comparison to the overlay to determine if it needs adjustments.

Damian3395 gravatar imageDamian3395 ( 2016-06-14 17:43:05 -0600 )edit

With this level of specificity all I can say is you need to either make assumptions, or change assumptions. So assume a known aspect ratio and try different thresholds until you find it. Or flip the image intensity so white is black and run your algorithm again.

Tetragramm gravatar imageTetragramm ( 2016-06-14 18:17:39 -0600 )edit