Ask Your Question
0

Muct Dataset Memory Overflow.

asked 2014-03-06 03:39:57 -0500

MRDaniel gravatar image

Hi.

I am using the MUCT dataset.

nSamples = 5828 nPixels = 90267 Total = 526076076 double cv::Mat AllTextures = cv::Mat::zeros(nSamples, nPixels, CV_64FC1);

Terminal Error. OpenCV Error: Insufficient memory (failed to allocate 4208612 bytes) in cv::OutOfMemoryError.

Visual Studio Dialog "Unhandled exception at at 0x75BBC41F in OpencvAAM.exe: Microsoft C++ exception: cv::Exception at memory location 0x003CCBE4."

Is there anything i can do about this other than cut back on data samples? Ultimately, this data needs to be handled, with PCA etc, so there will be more data further down the pipeline?

System Information

OpenCV 2.4.8.

Visual Studio 2012

Windows 7

System RAM 4.0Gb

Virtual Memory 4567Mb

edit retag flag offensive close merge delete

Comments

The cropped, flipped/clone and resized MUCT dataset of 5828 images is only 156Mb on disk.

flipped/clone is used to double dataset size.

MRDaniel gravatar imageMRDaniel ( 2014-03-06 04:25:38 -0500 )edit

Which version of Win 7 do you use? 32 bit or 64 bit? Because your AllTextures Mat object requires a huge memory area: 526076076 double = 526076076*8 /(1024x1024x1024) = 3.92 Gb and in Windows 7 32 bit, by default, a process (of a program) can only allocate and use 2 Gb of memory.

tuannhtn gravatar imagetuannhtn ( 2014-03-06 05:17:02 -0500 )edit

It's definitely 64bit Windows 7 Home Premium. I will be upgrading soon, but in the meantime i was hoping there was a work around. Is there a way to do PCA cumulatively?

MRDaniel gravatar imageMRDaniel ( 2014-03-06 05:49:58 -0500 )edit

Alright, in any case, it is a "out of memory" error, since your matrix ~ amount of system RAM, it can not be allocated (beside your program, Windows must keep some system programs, about 500 Mb->1Gb inside the RAM to work). I do not know a cumulative PCA algorithm, but with such huge matrix, the computation time must be very long. Just make sure you compile your program with x64 build mode because by default VS's 32 bit build mode will generate a program with a barrier of 2 Gb memory.

tuannhtn gravatar imagetuannhtn ( 2014-03-06 06:19:40 -0500 )edit

It will take a long time, hopefully it'll be worth it. :) I'll try the x64 setting and let you know.

MRDaniel gravatar imageMRDaniel ( 2014-03-06 06:43:38 -0500 )edit

Hmmm.... any clues here?

Error 37 error C1189: #error : "No Target Architecture" C:\Program Files (x86)\Windows Kits\8.0\Include\um\winnt.h 146 1 OpencvAAM

IntelliSense: #error directive: "No Target Architecture" c:\Program Files (x86)\Windows Kits\8.0\Include\um\winnt.h 146 2 OpencvAAM

IntelliSense: identifier "PCONTEXT" is undefined c:\Program Files (x86)\Windows Kits\8.0\Include\um\minwinbase.h 357 9 OpencvAAM

I'm getting an issue with WinBase.h and winnt.h and processthreadsapi.

http://forums.codeguru.com/showthread.php?460708-RESOLVED-PCONTEXT-undefined

I am using OpenCV 2.4.8 from Nuget. I am not including windef,h or winbase.h anywhere.

If the 2Gb cap on memory can be surpassed then i'm in a little bit of trouble here.

MRDaniel gravatar imageMRDaniel ( 2014-03-06 06:49:53 -0500 )edit

With x64 mode build, try do not use #include <Winuser.h> and suggestions from http://stackoverflow.com/questions/257134/weird-compile-error-dealing-with-winnt-h.

With 32 bit build mode, use instructions from http://msdn.microsoft.com/en-us/library/wz223b1z(v=vs.110).aspx, your 32bit program can manage more than 2 Gb memory.

tuannhtn gravatar imagetuannhtn ( 2014-03-06 07:16:34 -0500 )edit

Ok. Using a 12Gb RAM machine. It is a 64bit machine, but it is a 32bit version of visual studio (Express).

We get past the issue of the allocation for the PCA datamatrix, and PCA proceeds for some time.

cv::PCA pca = cv::PCA(AllTextures, cv::noArray(), CV_PCA_DATA_AS_ROW);

then, at some point within the black box that is PCA 104820072 bytes - cv::OutOfMemoryException Which is around 99Gb.

Seriously, there must be a way to predict datasizes for PCA otherwise, we try for 12-20 hours and get an exception? That seems somewhat unwise and i would even say it's a bug in the functionality.

MRDaniel gravatar imageMRDaniel ( 2014-03-11 23:59:45 -0500 )edit

2 answers

Sort by ยป oldest newest most voted
0

answered 2014-03-12 12:36:59 -0500

xaffeine gravatar image

Building for 64-bit is the way to enable your program to use more than 4GB of memory.

Independent of that, if PCA is now your first step, you could precede it with DCT on each image and keep a fraction of the resulting coefficients. You will save memory and gain speed this way.

edit flag offensive delete link more
0

answered 2014-03-12 00:31:18 -0500

updated 2014-03-12 00:33:27 -0500

No, 104820072 bytes are just about 99 Mb, not Gb. AFAIK, OpenCv is not famous for that kind of task, PCA, especially with big data.I suggest you to use a feature extraction algorithm before doing the PCA to reduce the size of the matrix. And what are you trying to do with Muct?

edit flag offensive delete link more

Comments

I'm creating an AAM. I've started again, trying to reduce the number of vectors retained. Hoping this 99Mb is the final output being initialised. It works for a smaller set and retaining less values should hopefully help. Fingers crossed.

MRDaniel gravatar imageMRDaniel ( 2014-03-12 02:35:47 -0500 )edit
Login/Signup to Answer

Question Tools

Stats

Asked: 2014-03-06 03:39:57 -0500

Seen: 659 times

Last updated: Mar 12 '14