Muct Dataset Memory Overflow.
Hi.
I am using the MUCT dataset.
nSamples = 5828 nPixels = 90267 Total = 526076076 double cv::Mat AllTextures = cv::Mat::zeros(nSamples, nPixels, CV_64FC1);
Terminal Error. OpenCV Error: Insufficient memory (failed to allocate 4208612 bytes) in cv::OutOfMemoryError.
Visual Studio Dialog "Unhandled exception at at 0x75BBC41F in OpencvAAM.exe: Microsoft C++ exception: cv::Exception at memory location 0x003CCBE4."
Is there anything i can do about this other than cut back on data samples? Ultimately, this data needs to be handled, with PCA etc, so there will be more data further down the pipeline?
System Information
OpenCV 2.4.8.
Visual Studio 2012
Windows 7
System RAM 4.0Gb
Virtual Memory 4567Mb
The cropped, flipped/clone and resized MUCT dataset of 5828 images is only 156Mb on disk.
flipped/clone is used to double dataset size.
Which version of Win 7 do you use? 32 bit or 64 bit? Because your AllTextures Mat object requires a huge memory area: 526076076 double = 526076076*8 /(1024x1024x1024) = 3.92 Gb and in Windows 7 32 bit, by default, a process (of a program) can only allocate and use 2 Gb of memory.
It's definitely 64bit Windows 7 Home Premium. I will be upgrading soon, but in the meantime i was hoping there was a work around. Is there a way to do PCA cumulatively?
Alright, in any case, it is a "out of memory" error, since your matrix ~ amount of system RAM, it can not be allocated (beside your program, Windows must keep some system programs, about 500 Mb->1Gb inside the RAM to work). I do not know a cumulative PCA algorithm, but with such huge matrix, the computation time must be very long. Just make sure you compile your program with x64 build mode because by default VS's 32 bit build mode will generate a program with a barrier of 2 Gb memory.
It will take a long time, hopefully it'll be worth it. :) I'll try the x64 setting and let you know.
Hmmm.... any clues here?
Error 37 error C1189: #error : "No Target Architecture" C:\Program Files (x86)\Windows Kits\8.0\Include\um\winnt.h 146 1 OpencvAAM
IntelliSense: #error directive: "No Target Architecture" c:\Program Files (x86)\Windows Kits\8.0\Include\um\winnt.h 146 2 OpencvAAM
IntelliSense: identifier "PCONTEXT" is undefined c:\Program Files (x86)\Windows Kits\8.0\Include\um\minwinbase.h 357 9 OpencvAAM
I'm getting an issue with WinBase.h and winnt.h and processthreadsapi.
http://forums.codeguru.com/showthread.php?460708-RESOLVED-PCONTEXT-undefined
I am using OpenCV 2.4.8 from Nuget. I am not including windef,h or winbase.h anywhere.
If the 2Gb cap on memory can be surpassed then i'm in a little bit of trouble here.
With x64 mode build, try do not use #include <Winuser.h> and suggestions from http://stackoverflow.com/questions/257134/weird-compile-error-dealing-with-winnt-h.
With 32 bit build mode, use instructions from http://msdn.microsoft.com/en-us/library/wz223b1z(v=vs.110).aspx, your 32bit program can manage more than 2 Gb memory.
Ok. Using a 12Gb RAM machine. It is a 64bit machine, but it is a 32bit version of visual studio (Express).
We get past the issue of the allocation for the PCA datamatrix, and PCA proceeds for some time.
cv::PCA pca = cv::PCA(AllTextures, cv::noArray(), CV_PCA_DATA_AS_ROW);
then, at some point within the black box that is PCA 104820072 bytes - cv::OutOfMemoryException Which is around 99Gb.
Seriously, there must be a way to predict datasizes for PCA otherwise, we try for 12-20 hours and get an exception? That seems somewhat unwise and i would even say it's a bug in the functionality.