How to replace an instance with cv::Mat members with another instance via pointer?

edit

I'm doing online destructive clustering (clusters replace clustered objects) on a list of class instances (stl::list<percepunit>).

Background

My list of current percepUnits is: stl::list<percepUnit> units; and for each iteration I get a new list of input percepUnits stl::list<percepUnit> scratch; that need to be clustered with the units.

I want to maintain a fixed number of percepUnits (so units.size() is constant), so for each new scratch percepUnit I need to merge it with the nearest percepUnit in units. Following is a code snippet that builds a list (dists) of structures (percepUnitDist) that contain pointers to each pair of items in scratch and units percepDist.scratchUnit = &(*scratchUnit); and percepDist.unit = &(*unit); and their distance. Additionally, for each item in scratch I keep track of which item in units has the least distance minDists.

// For every scratch percepUnit:
for (scratchUnit = scratch.begin(); scratchUnit != scratch.end(); scratchUnit++) { 
    float minDist=2025.1172; // This is the max possible distance in unnormalized CIELuv, and much larger than the normalized dist.
    // For every percepUnit:
    for (unit = units.begin(); unit != units.end(); unit++) { 

        // compare pairs
        float dist = featureDist(*scratchUnit, *unit, FGBG);
        //cout << "distance: " << dist << endl;

        // Put pairs in a structure that caches their distances
        percepUnitDist percepDist;
        percepDist.scratchUnit = &(*scratchUnit); // address of where scratchUnit points to.
        percepDist.unit = &(*unit);
        percepDist.dist = dist;

        // Figure out the percepUnit that is closest to this scratchUnit.
        if (dist < minDist)
            minDist = dist;

        dists.push_back(percepDist); // append dist struct
    }
    minDists.push_back(minDist); // append the min distance to the nearest percepUnit for this particular scratchUnit.
}

So now I just need to loop through the percepUnitDist items in dists and match the distances with the minimum distances to figure out which percepUnit in scratch should be merged with which percepUnit in units. The merging process mergePerceps() creates a new percepUnit which is a weighted average of the "parent" percepUnits in scratch and units.

My understanding is that the following code should do the job just fine, but for some reason memory usage increases with each of these "merge" operations. Even if I provide a method that calls clone() for cv::Mat members, I still get an increase in memory. If I don't copy the cv::Mats at all there is no unexpected memory increase. If I don't "replace" the old instance with the newly merged one, I also don't see memory increase. This is related to this question

// Loop through dists and merge all the closest pairs.
// Loop through all dists
for (distIter = dists.begin(); distIter != dists.end(); distIter++) {
    // Loop through all minDists for each scratchUnit.
    for (minDistsIter = minDists.begin(); minDistsIter != minDists.end(); minDistsIter++) {
        // if this is the closest cluster, and the closest cluster has not already been merged, and the scratch has not already been merged.
        if (*minDistsIter == distIter->dist and not distIter->scratchUnit->remove) {

            percepUnit newUnit;
            mergePerceps(*(distIter->scratchUnit), *(distIter->unit), newUnit, FGBG);
            *(distIter->unit) = newUnit; // replace the cluster with the new merged version.

            distIter->scratchUnit->remove = true;
        }
    }
}

For completeness, here is the whole percepUnit ... (more)