I cannot load my training descriptors in memory

asked 2015-04-10 17:15:33 -0500

Abu Gaseem gravatar image

updated 2015-04-10 17:54:47 -0500

I'm working on Bag of Visual words module .I extract SIFT descriptors from 984 Training images,and stored them on disk as .yaml file . I wonder why my descriptors file .yaml is too huge ,it's size 923MB ,although the size of my whole training images is just 146MB . My problem that i cannot load my descriptors file which is a matrix has 1.4 million rows and 128 cols in memory it's look there's a threshold used in the opencv alloc.cpp prevent me from allocate more than 730MB . is it normal that i the descriptors file .yaml is too huge according to the training image size which is 146MB ?

edit retag flag offensive close merge delete

Comments

1

the size of the descriptors Mat is unrelated to the size of the images

berak gravatar imageberak ( 2015-04-11 01:14:06 -0500 )edit

how i can load my descriptor file ? I think i would try to implement serialize and desalinize into a txt system file .

Abu Gaseem gravatar imageAbu Gaseem ( 2015-04-11 02:16:25 -0500 )edit
1

yes, most likely, you will have to come up with your own serialization.

the problem with the yaml / xml FileStorage is, that they have to read the whole file in, before they can start parsing it, you probably can avoid that with your own solution.

but again, do you really need to store the descriptors ? the BOW training is an offline, one-time step, what you need to save is the resulting (much shorter) vocabulary Mat, after that, you'll never need those descriptors again.

berak gravatar imageberak ( 2015-04-11 02:35:35 -0500 )edit
1

Yes you are 100% right , so that i dont save them just go to the building the dictionary streadforward and now the clustering is running . just one question , i wonder why the BOWTrainer.cluster() method is using my 4 cores , is it normal (i.e is it multithreaded computaion) ? provided that i enable openmp during compilation my program .

Abu Gaseem gravatar imageAbu Gaseem ( 2015-04-11 03:01:50 -0500 )edit
1

kmeans should use parallel threads, if you enabled it.

berak gravatar imageberak ( 2015-04-11 03:26:00 -0500 )edit

I enabled openMp during building the opencv lib 2.4.9. also in my application i enabled the openMp flags .

Abu Gaseem gravatar imageAbu Gaseem ( 2015-04-11 03:33:40 -0500 )edit