Hello all,
I've been working on a project for a while not realizing I've been creating a lot of memory leaks as I had not run the program for long enough until recently. It's a fairly large program, so I'll provide a single example that should describe the overall problem.
Firstly, my image data is stored in a class (percepUnit) which has a number of cv::Mat members:
class percepUnit {
public:
cv::Mat image; // percept itself
cv::Mat mask; // alpha channel
cv::Mat alphaImage; // mask + image.
// Create RGBA image from RGB+Mask
cv::Mat applyAlpha(cv::Mat image, cv::Mat mask);
}
I want to apply the mask to the image to create the alpha image. This is what the class method looks like:
// Apply the mask as an alpha channel
cv::Mat percepUnit::applyAlpha(cv::Mat image, cv::Mat mask) {
vector<cv::Mat> channels;
cv::Mat alphaImage;
if (image.rows == mask.rows and image.cols == mask.cols) {
cv::split(image,channels); // break image into channels
channels.push_back(mask); // append alpha channel
cv::merge(channels,alphaImage); // combine channels
}
return alphaImage;
}
Which is used in the percepUnit constructor:
percepUnit::percepUnit(cv::Mat ROI, cv::Mat alpha, int ix, int iy, int iw, int ih, int area) {
// Deep copies.
image = ROI.clone();
mask = alpha.clone();
// Make alpha image
// there may be a more efficient way of doing this. (drawMat() for RGBA?)
this->alphaImage = applyAlpha(image, mask);
After some searching it does seem like returning cv::Mats is not a good idea.
This is the valgrind output:
==25374== 13,422,700 bytes in 21 blocks are possibly lost in loss record 17,975 of 17,982
==25374== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25374== by 0x33C0BA90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==25374== by 0x33C51BF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==25374== by 0x33C7313B: cv::_OutputArray::create(int, int const*, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==25374== by 0x33D98DA7: cv::merge(cv::Mat const*, unsigned long, cv::_OutputArray const&) (in /usr/local/lib/libopencv_core.so.2.4.5)
==25374== by 0x33D999D0: cv::merge(cv::_InputArray const&, cv::_OutputArray const&) (in /usr/local/lib/libopencv_core.so.2.4.5)
==25374== by 0x33D99ABD: cv::merge(std::vector<cv::Mat, std::allocator<cv::Mat> > const&, cv::_OutputArray const&) (in /usr/local/lib/libopencv_core.so.2.4.5)
==25374== by 0x415A95: percepUnit::applyAlpha(cv::Mat, cv::Mat) (percepUnit.cpp:19)
==25374== by 0x415C73: percepUnit::percepUnit(cv::Mat, cv::Mat, int, int, int, int, int) (percepUnit.cpp:212)
Should I rewrite the applyAlpha method to take the cv::Mat &alphaImage as an argument and forgo the return statement?
EDIT: Here is my new applAlpha function:
// Apply the mask as an alpha channel
void percepUnit::applyAlpha(const cv::Mat &image, const cv::Mat &mask, cv::Mat &alphaImage) {
// Avoid merge, helps with memory leak?
cv::Mat src[] = {image, mask};
int from_to[] = {0,0, 1,1, 2,2, 3,3};
alphaImage = Mat(image.rows, image.cols, CV_8UC4);
cv::mixChannels(src, 2, &alphaImage, 1, from_to, 4); // &(*alphaImage)?
}
And the resulting Valgrind output:
==30629== 15,140,804 bytes in 27 blocks are possibly lost in loss record 17,993 of 17,999
==30629== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30629== by 0x33C0BA90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==30629== by 0x33C51BF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==30629== by 0x415EF6: cv::Mat::create(int, int, int) (mat.hpp:353)
==30629== by 0x415083: percepUnit::applyAlpha(cv::Mat const&, cv::Mat const&, cv::Mat&) (mat.hpp:75)
==30629== by 0x415B44: percepUnit::percepUnit(cv::Mat, cv::Mat, int, int, int, int, int) (percepUnit.cpp:205)
Will there always be some small leaks in this function?
Things have improved enough that now the second worst leak in the program is on top. I did not want to start with this one because it was not as serious, and its a quite a bit more complex. I'm only posting now because its a last resort and I have no idea how to approach it.
Here we have a simple class to store some metaData along with a Mat:
// class to store background masks with boundingBoxes from floodFill
class backgroundMask{
public:
cv::Mat mask;
cv::Rect bounds;
int area;
backgroundMask(cv::Mat &argMask, cv::Rect &argRect, int argArea);
~backgroundMask();
};
The constructor and destructor:
// class to store background masks with boundingBoxes from floodFill
// TODO change this into a struct or remove entirely since we resegment every frame now?
backgroundMask::backgroundMask(Mat &argMask, Rect &argRect, int argArea) {
this->mask = argMask.clone(); // try and avoid memory leaks.
//argMask.copyTo(mask);
bounds = argRect;
area = argArea;
}
backgroundMask::~backgroundMask() {
mask.release();
}
Instances of these are stored in an stl::list: list<backgroundMask>
Which is populated in a function like so:
// Process one frame, write to masks list.
// TODO 27% CPU here.
list<backgroundMask> segmentation::backgroundMasks(Mat &image) {
Mat mask, maskTrack;
list<backgroundMask> masks;
Rect *boundingRect = new Rect(); // Stored bounding box for each flooded area.
Rect transBoundingRect; // transformed bounding rect correcting for downsampling
// do some processing on image
//creates blur, maskThresh, newX, newY, newW, newH.
// Segmentation using floodFill
maskTrack = Mat( blur.rows+2, blur.cols+2, CV_8UC1, Scalar::all(0) ); // Make black single-channel image.
mask = Mat( blur.rows+2, blur.cols+2, CV_8UC1, Scalar::all(0) ); // image for individual masks.
// Loop through all the pixels and flood fill.
for( int y = 0; y < blur.rows; y++ )
{
for( int x = 0; x < blur.cols; x++ )
{
if( maskTrack.at<uchar>(y+1, x+1) == 0 and x+1 <= maskTrack.rows and y+1 <= maskTrack.cols) // mask is offset from original image.
{
mask = Mat::zeros( mask.rows, mask.cols, CV_8UC1 ); // Clear for next floodFill WAS: mask.setTo(Scalar::all(0)) 20% CPU!
area = floodFill(blur, mask, Point(x,y), Scalar(), boundingRect, Scalar::all(1), Scalar::all(1), FLOODFILL_MASK_ONLY|8|(255<<8));
if (area > 75) {
transBoundingRect = Rect(newX, newY, newW, newH);
// append to masks list.
masks.push_back(backgroundMask(maskThresh,transBoundingRect, area)); // clone() too slow?
maskTrack += mask; // Keep track of floodFills.
numROIs++;
}
}
}
}
return masks;
}
The function is called like so:
masks.clear(); // make any difference to memory leak?
masks = backgroundMasks(backgroundScaled);
Here is the valgrind output:
==30629== 70,824,732 bytes in 33 blocks are possibly lost in loss record 17,998 of 17,999
==30629== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30629== by 0x33C0BA90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==30629== by 0x33C51BF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==30629== by 0x33C52378: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==30629== by 0x33CD6F7D: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==30629== by 0x4163FB: backgroundMask::backgroundMask(cv::Mat&, cv::Rect_<int>&, int) (mat.hpp:335)
The Mat::copyTo() is confusing, because I switched to using Mat::clone().
Sorry for the extra problem, but I thought the answer to the applyAlpha problem would make this one clear. The size of the leak seems to indicate Mats are being leaked, but it could be some of the other structures, so I've included those here too.
EDIT 2
I stepped away from the problem for a few weeks preparing for and attending a conference, and I was unable to solve the memory leak problem. Valgrind is still telling me that the applyAlpha function is "possibly" leaking memory:
==7825== 767,228,144 bytes in 452 blocks are possibly lost in loss record 18,040 of 18,040
==7825== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7825== by 0x33C0BA90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==7825== by 0x33C51BF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==7825== by 0x415E66: cv::Mat::create(int, int, int) (mat.hpp:353)
==7825== by 0x414FF3: percepUnit::applyAlpha(cv::Mat const&, cv::Mat const&, cv::Mat&) (mat.hpp:75)
==7825== by 0x415AB4: percepUnit::percepUnit(cv::Mat, cv::Mat, int, int, int, int, int) (percepUnit.cpp:205)
==7825== by 0x41C996: clustering::mergePerceps(percepUnit, percepUnit, std::string) (clustering.cpp:250)
==7825== by 0x41D6E5: clustering::cluster(std::list<percepUnit, std::allocator<percepUnit> >&, std::list<percepUnit, std::allocator<percepUnit> >&, std::string) (clustering.cpp:428)
==7825== by 0x41FA93: segmentationClusteringThread::tick() (segmentationClusteringThread.cpp:356)
==7825== by 0x41FF41: segmentationClusteringThread::threadedFunction() (segmentationClusteringThread.cpp:425)
==7825== by 0x501AD2: ofThread::run() (in /home/bbogart/Projects/svn/Dreaming-Machine-3/trunk/DM/bin/DM)
==7825== by 0x578E52: Poco::ThreadImpl::runnableEntry(void*) (in /home/bbogart/Projects/svn/Dreaming-Machine-3/trunk/DM/bin/DM)
Since this leak is "possible" could this be a false leak? Could my real leak (shown in the increase of allocated memory for the whole program, and its eventual crash due to allocating more memory than is available) is coming from somewhere else, perhaps leaking much less? The problem is the valgrind output is huge, and I have just been looking at the highest numbers of possible leaking memory. I don't know how to approach the debugging process to narrow down this leak. Another valgrind entry with lots of possible leaking memory is the following:
==7825== 575,424,272 bytes in 452 blocks are possibly lost in loss record 18,039 of 18,040
==7825== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7825== by 0x33C0BA90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==7825== by 0x33C51BF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==7825== by 0x33C52378: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==7825== by 0x33CD6F7D: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==7825== by 0x415A0E: percepUnit::percepUnit(cv::Mat, cv::Mat, int, int, int, int, int) (mat.hpp:335)
==7825== by 0x41C996: clustering::mergePerceps(percepUnit, percepUnit, std::string) (clustering.cpp:250)
==7825== by 0x41D6E5: clustering::cluster(std::list<percepUnit, std::allocator<percepUnit> >&, std::list<percepUnit, std::allocator<percepUnit> >&, std::string) (clustering.cpp:428)
==7825== by 0x41FA93: segmentationClusteringThread::tick() (segmentationClusteringThread.cpp:356)
==7825== by 0x41FF41: segmentationClusteringThread::threadedFunction() (segmentationClusteringThread.cpp:425)
==7825== by 0x501AD2: ofThread::run() (in /home/bbogart/Projects/svn/Dreaming-Machine-3/trunk/DM/bin/DM)
==7825== by 0x578E52: Poco::ThreadImpl::runnableEntry(void*) (in /home/bbogart/Projects/svn/Dreaming-Machine-3/trunk/DM/bin/DM)
Which is the constructor to create new percepUnits, which is described in my original question above.
Should I ignore these huge "possible" losses and just look at the "definitely" lost (still reachable)? Maybe all this time I've been looking in the wrong place for the leak...
I use a lot of lists and vectors to store instances. I know if I use "new" then I need to "delete" to prevent a leak, but what if I just call a function that returns an instance? For example:
masks.push_back(backgroundMask(maskThresh,transBoundingRect, area));
Do I need to do anything special?
Is this different than:
backgroundMask* tmpMask;
...
tmpMask = new backgroundMask(maskThresh,transBoundingRect, area);
masks.push_back(tmpMask);
...
delete tmpMask;
I've only been seriously using C++ for the last year and lack any formal training in it...
