Most efficient format to save and load file
Hi all,
I'm wondering if YML/XML is the most efficient format to save and load file in OpenCV. The situation is that I'm gonna have a really big SIFT features matrix (say, 10 million features, which means 10 million rows and 128 columns), and I want to load (read from disk) it in the fastest way possible. Slow saving is ok but reading need to be fast.
Thanks
the most efficient/fastest way to read write data in C++ is as binary files. However, I do not think that there is this option embedded in opencv. Therefore, if you still want to do it with opencv you will either have to go with
YML/XMLapproach or by using theformat()function as it is described here and save the data in another format lke.csv. and then try to read it again with the method described again in the above link or with another parser which might be faster. Actually, you could make some tests, measure the performance and tell us what is faster.hmmm, for such large things, i'd roll my own binary serialization using fopen() or such.
in fact, all you need to save to reconstruct a cv::Mat is rows,cols,type and data
using opencv's Filestorage has some serious drawbacks, e.g. for xml, a dom parser is used, meaning, it has to read the whole thing into memory, build the real model from that, and at that time you need like 2 x the memory for your model, which gets you close to e.g. 32bit limitations fast.
i did not test it but look at SO Answer may be it helps.
we definitely need to convince Miki, that SO is a rotten place, and that he should spread his wisdom here ;)
@berak count me in if you decide on a kidnapping plan :-p... @sturkmen nice hit ;-)
What about imwrite with any losless format
10 million SIFT (float) features.
I'm confused. what did you say about Miki's code ? i tried to do with imwrite what Miki did.
i think, Miki had the appropriate idea there.
think of it, - pgm ppm or such are just writing a (very short) text header with rows, cols, type, then followed by a large binary data block. pretty close to a plain fopen() approach (there won't be much difference with timimg)
unfortunately, your imwrite() example below restricts the type to CV_8U, while CV_32F would be required to save/load SIFT features.
so, imho, better shortcut any openv builtin functionality, which cannot handle this (atm).
OK. sorry it seems i didn't realize the question and Miki's code well.