OpenCV holds the memory longer than it is needed

edit

I have a this program:

Mat m1 = Mat::zeros(5000, 5000, CV_8U);
Mat m2 = Mat::zeros(5000, 5000, CV_8U);
Mat m3 = Mat::zeros(5000, 5000, CV_8U);
//...
Mat m10 = Mat::zeros(5000, 5000, CV_8U);
// use the mats ...

After this code block ends, the Mats are out of scope and there are definitely no other references to the data, the process somehow still take ten of megabytes of RAM. if I do this Instead:

uchar m1 = new uchar[5000 * 5000];
uchar m2 = new uchar[5000 * 5000];
uchar m3 = new uchar[5000 * 5000];
//...
uchar m10 = new uchar[5000 * 5000];

// use the mats ...

delete [] m1;
delete [] m2;
delete [] m3;
//...
delete [] m10;

the process takes almost zero memory after this code. I am guessing OpenCV has some internal memory management where it doesn't free the memory for potential future use. That's a problem for me because I need this memory for other things after this code run. This process serves are a server so it stays up indefinitely. Is there a way to force OpenCV to free this memory?

I am using OpenCV 4.3.0 on Ubuntu 16.04