Revision history [back]

Hi all

Thank you for your input/suggestion to my question how to determine the LCD/LED display in an image.

The figure 1 below is a scratchcard which we are interested to locate/segment based on the yellowish color part ,which i refer to as the LCD/LED display. After segmenting the LCD/LED area based on the Yellowish color we can extract the digits present

                 Figure 1 Scratchcard Image

For this this task we will use OpenCV and C++
OpenCV captures images in BGR (BLUE,RED and GREEN ) format, and not RGB ( RED, GREEN and BLUE) as one would expect.The captured images are 3 bytes ( 24 bits ) of data .The 3 bytes of data for each pixel is split into 3 different parts which are BLUE,RED and GREEN having 8-bits or 1 byte each. 1 byte can store a value from 0 to 255.This means that each BLUE having 256 variations of Blue , each GREEN having 256 variations of Green and each RED having 256 variations of Red.These primary colors can be mixed with different variations of each to get the desired color in this case yellowish or LCD/LED display color. Figure 2 shows the RBG colorspace.

Figure 2 RBG color space

In this task what is needed is to isolate the yellowish color in order to determine the LCD/LED display This is referred to as color based segmentation which is also know as With thresholding.However while OpenCV images are captured in BGR format ,The BGR format falls short for color based segmentation task. It seems using HSV color space shown in figure 3 will be more suitable.

Figure 3 HSV

HSV stands for Hue, Saturation, and Value.The Hue defines the color component ,Saturation defines how strong the color component is in other words how close that color is to white and Value defines the brightness of the color component or how close that color is to black..Therefore unlike RGB, HSV separates brightness in an image from the color information. This is very useful for this task at hand . Further it gives the the upper hand of having a single number color for the color of interest despite multiple shades of that color

in OpenCV the HSV values ranges are different from other application such as Gimp whose HSV ranges are: Hue ranges from 0 to 360, Saturation from 0 to 100 and Value from 0 to 100. while in OpenCV the HSV values ranges are Hue ranges from 0 to 180,Saturation ranges from 0 to 255, and Value ranges 0 to 255.

Gimp Hue values for the colors are : Orange 0-44 ,Yellow 44- 76, Green 76-150, Blue 150-260, Violet 260-320, Red 320-360

OpenCV Hue values for the colors Orange 0-22 Yellow 22- 38 Green 38-75 Blue 75-130 Violet 130-160 Red 160-180

For this task, the suitable ranges for the HSV ,after experimenting with different values to get the yellowish part , are Hue from ranges 20 to 70 , Saturation ranges 100 to 255, and Value ranges from 100 to 255.

Now lets go into the details of things the following is what the code does

1) loads the image source image

2) locates the lcd/led display

3) extract the digits from the LCD/LED display and pass it to tessaract for character recognition

Figure 4 below is a result of a figure 1 image which has been converted into HSV format for easy colour detection and threshold .The Yellow to green part as white and all other regions as black. Using the customised function cv::Mat GetThresholdedHSVImage( cv::Mat TempSourceImage )

                 Figure 4

The the source image in figure 1 is passed to the GetLcdImageArea customised function which returns the mask of the yellowish part.An and operation is operated on the return image to result the image in figure 5 below. Below are the operations

LcdSergement=GetLcdImageArea(SrcImage);

// Extract the RGB Lcd segement using logical operation AND:
LcdSergement &=SrcImage;

                 Figure 5

Figure 5 image is converted to to gray scale a Gaussian blur is used to smooth out the image. then adaptive thresholding to binarize the image so as to make it black or white and then invert the colors . OpenCV uses black as the background and white as the objects. // Convert source SrcImage from RBG to to GrayScale Image for later processing
cvtColor(SrcImage,GrayImage,CV_BGR2GRAY);

//Gaussian blur used to obtain a smooth grayscale image and also to reduce noise GaussianBlur(GrayImage,GrayImage,size,0);

// Use adaptiveThreshold Convert to binary to easily detect edges and to have // an even distribution illumination intensity

adaptiveThreshold(GrayImage,GrayImage,255, CV_ADAPTIVE_THRESH_MEAN_C, CV_THRESH_BINARY, 75, 10);

cv::bitwise_not(GrayImage,GrayImage); the above operations results image in figure 6 , however the image 6 has some noise or unwanted object/ contours.To remove the unwanted objects we pass it through the customized function DigitsOnly=RemoveSmallContours to remove small contours.

                  Figure 6

Figure 7 shows the resulting image after removing the small contours using the RemoveSmallContours(cv::Mat TempSourceImage) customised function

                 Figure 7

Figure 7 is Binarized Image which is passed to it to tesseract OCR customised function TesseractOCRResult which takes on the binarized image and converts the image to text

the resulted image is now text extracted from which is returned TesseractOCRResult function

52931 75003 3794

to compile the code below you will need to use the following

g++ -I/usr/local/include -L/usr/local/lib scratchcard_ocr.cpp -o scratchcard_ocr -lopencv_core -lopencv_imgproc -lopencv_highgui -ltesseract

Credit goes to Nash for this i got it from his blog

below is the source code

'#include "stdio.h"

'#include "math.h"

'#include "cv.h"

'#include "tesseract/baseapi.h"

'#include "iostream"

'#include "opencv2/imgproc/imgproc.hpp"

'#include "opencv2/highgui/highgui.hpp"

using namespace cv;

using namespace std;

const char* Tesseractlanguage = "eng"; // initialize Tesseract with english language training data const char* TesseractCharWhitelist = "tessedit_char_whitelist"; // limit tesseract to recognize only the wanted characters const char* TesseractDigitsOnly = "0123456789"; // Recognise Digits only

cv::Mat GrayImage; cv::Mat SrcImage; cv::Mat LcdImageArea; cv::Mat LcdSergement; cv::Mat DigitsOnly; cv::Size size(5,5);

void displayImages();

char* TesseractOCRResult(cv::Mat TempSourceImage);

cv::Mat GetLcdImageArea(cv::Mat TempSourceImage);

cv::Mat RemoveSmallContours(cv::Mat TempSourceImage);

cv::Mat GetThresholdedHSVImage(cv::Mat TempSourceImage );

int GetlargestContour(vector<vector<point> > contours);

int main(int argc, char** argv){

vector<vector<Point> > contours;
vector<vector<Point> > squares;

cv::Mat SrcImage = imread(argc == 2 ? argv[1] : "scratchcard_sample_image.jpg ");

if (SrcImage.empty()) 
      return -1;



 LcdSergement=GetLcdImageArea(SrcImage); 

cv::Mat lcdSergmentOnly=LcdSergement.clone();

 // Extract the RGB Lcd segement using logical operation AND:        
 LcdSergement &=SrcImage; 

 // Convert source SrcImage  from RBG to to GrayScale Image for later processing  
 cvtColor(SrcImage,GrayImage,CV_BGR2GRAY);   

 //Gaussian blur used  to obtain a smooth grayscale image and also to reduce noise
 GaussianBlur(GrayImage,GrayImage,size,0);

 //  Use adaptiveThreshold Convert to binary to  easily detect edges  and to have 
 // an even distribution illumination intensity 

 adaptiveThreshold(GrayImage,GrayImage,255,
                   CV_ADAPTIVE_THRESH_MEAN_C, 
                   CV_THRESH_BINARY,
                   75,
                   10);


  cv::bitwise_not(GrayImage,GrayImage);   

  cvtColor(lcdSergmentOnly,lcdSergmentOnly,CV_BGR2GRAY);

  //  Extract the gray Lcd segement using logical operation AND: 
  lcdSergmentOnly &=GrayImage;

  // Remove small contours that are considered to be noise
  DigitsOnly=RemoveSmallContours(lcdSergmentOnly);      

  // Pass the binarize gray scale image to tesseract for character recognition
  TesseractOCRResult(DigitsOnly);    

  // Display the Source Image 
  namedWindow("Source Image", 0 );
  imshow("Source Image",SrcImage);  

  //Display the LCD segement     
  namedWindow( "Lcd Image Area", 0 );
  imshow("Lcd Image Area",LcdSergement); 


  // Display the digits only from the image 
  namedWindow( "Retrived Digits", 0 );
  imshow("Retrived Digits",DigitsOnly); 


  // Wait untill a key is pressed to end application  
  waitKey(0);

  // distroy all Windows
  cvDestroyAllWindows();

}

// This function take RGB image.uses the GetThresholdedHSVImage () to covnert to into HSV
// Processes Blur/smoothing operation on the image find cont. // Then return that image. cv::Mat GetLcdImageArea(cv::Mat TempSourceImage){

  vector<vector<Point> > contours;

  LcdImageArea=GetThresholdedHSVImage(TempSourceImage);    

  blur( LcdImageArea, LcdImageArea, Size(5,5) );      

 // find contours and store them all as a list contours
 findContours(LcdImageArea.clone(), 
              contours,  
              CV_RETR_EXTERNAL, 
              CV_CHAIN_APPROX_NONE);



 //initialize the LcdArea Image
 cv::Mat LcdArea= Mat::zeros(TempSourceImage.size(), TempSourceImage.type());

 drawContours(LcdArea, 
              contours,
              GetlargestContour(contours), //use GetlargestContour function to loop through all the contours and return  index of largest contour  
              Scalar::all(255), 
              -1);


return LcdArea;

}

// This function take RGB image.Then convert it into HSV for easy colour detection // and threshold it with yellow to green part as white and all other regions as black. // Then return that image. cv::Mat GetThresholdedHSVImage( cv::Mat TempSourceImage ){

cv::Mat HSVImage; 
cv::Mat ThresholdImage; 

// Convert source SrcImage  from RBG to to HSV Image foreasy colour detection 
cvtColor( TempSourceImage, HSVImage, CV_BGR2HSV );  

// Create binary thresholded image  to max/min HSV ranges  
// For detecting yellow to green LCD Area in  the Image - HSV mode  

    inRange( HSVImage,  
             Scalar( 20,  100, 100 ), //Minimum HSV range            
             Scalar( 70,  255, 255 ), //Maximum HSV range  
             ThresholdImage );  


return ThresholdImage;

}

// This function Loops through all the Contours to get the largest // and returns the index of the Largest Contour. int GetlargestContour(vector<vector<point> > contours) {

 double MaxArea     = 0;
 int    MaxIndex    = 0;      
 int    ContourSize = contours.size();

 for (int i = 0; i < ContourSize; i++) {    

      if (MaxArea  < cv::contourArea(contours[i])) {
          MaxArea  = cv::contourArea(contours[i]);
          MaxIndex = i;             
     }
 }

return MaxIndex;

} // This function take in a Gray Scale image finds Contours of the Image. // Loops through all the contours and only draws contours bellow a specified area size. // and returns the processed image cv::Mat RemoveSmallContours(cv::Mat TempSourceImage) {

  vector<vector<Point> > contours;

  cv::Mat SmallContoursRemoved = TempSourceImage.clone(); ;

  // find contours and store them all as a list contours
      cv::findContours(SmallContoursRemoved.clone(), 
                   contours, 
                   CV_RETR_EXTERNAL, 
                   CV_CHAIN_APPROX_NONE);

  int ContourSize=contours.size();


  for (int i = 0; i < ContourSize; i++) {

    if (cv::contourArea(contours[i]) < 100)

        cv::drawContours(SmallContoursRemoved, 
                         contours, i, 
                         Scalar::all(0),
                         CV_FILLED);        

  }     



return SmallContoursRemoved;

}

// This function take in Binarized Image passes it to tesseract OCR. // and. returns the converted text from the image // all in all converts image to text. char* TesseractOCRResult(cv::Mat TempSourceImage) {

tesseract::TessBaseAPI TessOCR;
//Initialize Tesseract to only english training data
TessOCR.Init(NULL, Tesseractlanguage );    

// Specifiy the Tesseract whitelist to only look for digits
TessOCR.SetVariable(TesseractCharWhitelist, TesseractDigitsOnly); 

TessOCR.SetPageSegMode(tesseract::PSM_SINGLE_BLOCK);

 // Pass the binarized image to tesseract 
TessOCR.SetImage((uchar*)TempSourceImage.data, 
                         TempSourceImage.cols,
                         TempSourceImage.rows, 
                         1, 
                         TempSourceImage.cols);

char* PinNumber = TessOCR.GetUTF8Text();
std::cout << PinNumber << std::endl;  


return PinNumber;

}