memory consumption while training > 50GB

I try to train opencv to recognize a licenseplate. I have prepared some samples (few thousands), prepared vec file successfully and run opencv_traincascade (parameters below). After few minustes i had to stop the process due to the memory consumption that hit 16GB of my RAM and more than 50BG of (dynamic) swap. I have tried to run it with fewer examples, lowered the resolution of negative images. I last try runs were as below but without success (the same memory consumption).

opencv_createsamples -vec positive.vec -bg negative/IMG_5596.JPG -num 200 -maxxangle 0.4 -maxyangle 0.4 -maxzangle 0.2 -w 533 -h 117 -info info.dat

opencv_traincascade -data result -vec info.dat -numStages 200 -w 533 -h 117 -bg bg.txt


(If needed i can provide all files i use to train opencv). Neither precalcValBufSize nor precalcIdxBufSize did't help.

I use OpenCV 2.4.5, build it myself from sources, MBP 2013, OSX 10.8.4

All suggestions what I may be doing wrong or how to train opencv appreciated. Thanks in advance.

edit retag close merge delete

1

i'm no expert on this, but -w 533 -h 117 seems quite a lot. e,g haarcascade_frontalface_alt.xml uses <size>20 20</size>

also, you're planning to train for 200 stages ? the universe will collapse, before that's ready ;)

( 2013-07-04 12:59:14 -0500 )edit

200 stages is a lot and in fact that was not my first attempt. I'v just tried to change parameters to get an idea which leads to such extraordinary memory consumption... I also experiment as this is my first attempt to use OpenCV and image recognition library :)

I thought about changing the samples size but i found no hint about relationship of memory and size. I'll give it a try.

Thanks for the response.

( 2013-07-04 14:17:53 -0500 )edit
1

honestly, finding good params for this is hard, you're not the first to struggle with it, and you won't be the last ..

( 2013-07-04 14:27:08 -0500 )edit
1

When it works (at least for the training part) it will be nice to explore many variants.

You were right about the size, using samples 226x61 made swapfiles grow only to 10GB which is acceptable... It's a pity that it is not clearly mentioned in the manual (the difference is incredible) however...

===== TRAINING 0-stage ===== <BEGIN OpenCV Error: Assertion failed (_img.rows * _img.cols == vecSize) in get, file /Users/abankowski/Downloads/opencv-2.4.5/apps/traincascade/imagestorage.cpp, line 154 libc++abi.dylib: terminate called throwing an exception Abort trap: 6

As this is different issue let's assume that the first problem is solved. ;)

( 2013-07-04 14:55:53 -0500 )edit

Sort by » oldest newest most voted

It may not be an answer as such but its too long for a comment. So here it goes.

CreateSamples

You can actually use this utility in four ways. You can use it for :

• Creating Distorted Training samples - Perspective Transform , Intensity Variation, etc.
• Undistorted Training samples - Just a rectangular crop, incase you have too many samples already
• Annotated Test Samples - Useful for testing performance of your cascade after training
• And to see whats there in a Vec file (provided you know the size of the samples used to create it) - Useful for seeing and verifying what you really want is there in it.

It's pretty much covered in the OpenCV Documentation. Let me point you to another link. One of the best and detailed tutorial for HAAR training and can be easily extended to LBP and HOG with ease, thanks to OpenCV Developers

Regarding Size and to know why it blows up go check the haarfeatures.cpp in \$OpenCV_DIR/apps/traincascade/ and check the "void CvHaarEvaluator::generateFeatures()" and try calculating how many "sums" you are creating per image and add the size of integral images to that also. That should give you a fair idea about the requirement of memory (thats not the end actually. There are still other things occupying memory but this is the major chunk).

And the problem is it needs everything in the memory to get the best features within the window which can discriminate between the positives and negatives such that most of the positives are correctly classified (currentDetectionRate > minHitRate is achieved) and very little amount of negatives are mis-classified (currentFalseAlarmRate < maxFalseAlarmRate is achieved) and a suitable threshold is set such that all conditions are satisfied in that stage and continues to the next stage till global requirements are met. That's the naive definition for the working of cascade classification. It continues adding features to the stage so that the defined Rates are achieved and moves on to the next stage. The training stops if defined maxFalseAlarmaRate ^ numStages is achieved before required stages. (In your case, for numstages = 200 for minFA = 0.5, its 10^-61) or numstages is reached.

And well, the following is obvious. You test your classifier. It slides a defined window(533 x 117 in your case) and looks at the location defined by the stage classifier, computes HAAR, > stage threshold ? test next stage : reject the window and slide to the next position and blah blah blah. It is all coded up in the facedetect.cpp in samples/c and you just have to call it by giving the cascade name.

Phew, that was long. Hope I didnt go wrong in the middle. (Please edit it if I have)

Things to note - Use small sample size for creating samples. Try LBP and HOG too. Go through that tutorial above (read it completely atleast once. Note: It is written for HAARTRAINING,the old buddy before traincascade. Make sure you take care of the changes). You must also not forget ...

more

Nice explanation btw! I am considering writing a decent guide with @Guanta on cascade training. We have discussed it before, and will take this information along when writing it!

( 2013-07-05 04:46:37 -0500 )edit
1

Actually, It will be very helpful to have a single guide because all the guides presently available online are sort of incomplete and the tutorial i mentioned is the best one i came across so far. I don't think you guys need help, but If you do, consider me in. I am not a pro but would love to help. Cheers!

( 2013-07-05 05:16:26 -0500 )edit

Don't let the karma convince you that I am a pro, on the contrary, I have only been using OpenCV actively for my PhD for the last 11 months. I just made it one of my hobbies to help people out here. Its true that the documentation on cascade classification isn't as complete as it should be in documentation, lacking mainly insight in how to use the parameters and how to interpret them. It is a process of experience which should get bundled. Since I am doing a research project on this topic for my work, I have the opportunity to bundle experience rather easy, so I will be eagered to share it afterwards.

( 2013-07-05 06:23:05 -0500 )edit

Actually what happens is quite normal. Using such a large large resolution (w and h parameters), is actually not possible with the OpenCV implementation. Internally it stores all the integral images for the different scales of the scale pyramid, which would grow exponentially large with larger resolution.

Try to reduce your sample set to a smaller size, take for example a 10x smaller resolution and then just rescale your input data to match that. The size of your model only defines the smallest possible object that can be detected, not the larger ones, since these can be easily downsampled.

Another note, please do not use the createsamples tool for anything else than grouping input images. The tool creates unrealistic data, which results in training images that are actually impossible. It is better to get a training set of real images covering all possible situations that your number plate can be in. Yes this can be a quite exhaustive work, knowing that good numberplate detectors are easily trained on datasets with 100.000 images!

Also, creating a detector with 200 stages (not that it will ever reach there) is basically insane. It would take ages to go through all these stages just to say if a patch is a possible detection candidate.

I think you should look deeper into the traincascade documentation, search around on this forum a bit (since there have been TONS of topics before you discussing this) and try to get a feeling with correct parameters.

more

1

Thanks for the answer, basically it is what I supposed to hear - imo it's a pity that it is not written in the tutorial. Simple graph would be helpfull for the memory issue.

Thanks for the side notes, they may be very helpfull. I assume that many experiments are behind me.

The documentation, though extensive does not cover all cases, for the very beginner and one who does not want to dig into the source may leave many understatements. Tools also do not give very friendly feedback... eg "Assertion failed (_img.rows * _img.cols == vecSize)". I'm still wondering why i got it (id didnt have it for one templates set, now it happens).

Cheers.

( 2013-07-05 15:16:54 -0500 )edit
3

Hi Abankowski, I also encountered the error Assertion failed (_img.rows * _img.cols == vecSize) I realized that this was caused when there is a mismatch between the -w and -h passed into opencv_createsamples and those passed into opencv_traincascade. When I regenerated the positive vec file with matching -w and -h, it worked.

( 2013-07-17 19:29:52 -0500 )edit

This is indeed the problem! You should pass the correct parameters. These assertions are there to keep sure that other functions don't crash, giving you even weirder problems!

( 2013-07-18 02:21:37 -0500 )edit