OpenCV Q&A Forum - RSS feedhttp://answers.opencv.org/questions/OpenCV answersenCopyright <a href="http://www.opencv.org">OpenCV foundation</a>, 2012-2018.Tue, 13 Aug 2013 02:27:05 -0500linear least-squares for large-scale dense matrixhttp://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.
Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);Thu, 08 Aug 2013 04:48:58 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/Comment by StevenPuttemans for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18494#post-id-18494If this problem is solved, please use the answer functionality, add your own solution and accept it. This way the problem will appeared solved and people could reuse your solution on a later time.Fri, 09 Aug 2013 03:44:45 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18494#post-id-18494Comment by Moster for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18475#post-id-18475My c++ knowledge is not the best, you know. But I think your Matrices are getting allocated on the stack which is kind of small, but fast. You should maybe create them on the heap. So try it like this: Mat *A = new Mat(29030, 7809, CV_32FC1); Mat *B = new Mat(29030, 122, CV_32FC1);
I could of course be totally wrong, but at the moment you are trying to allocate 878 MByte on the stack :P
Fri, 09 Aug 2013 01:17:21 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18475#post-id-18475Comment by grtwa for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18476#post-id-18476Thank you. I have solved this problem by convert the compiling platform to x64. On x86 platform, only 2G physical memory can be used. It's my fault, not OpenCV :)Fri, 09 Aug 2013 01:28:51 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18476#post-id-18476Comment by grtwa for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18498#post-id-18498@Moster, I did not try because I think automatic variables are created on the heap. Any way, problem is gone :)Fri, 09 Aug 2013 04:31:07 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18498#post-id-18498Comment by Moster for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18482#post-id-18482Did you try my solution? Im curious if that actually works.Fri, 09 Aug 2013 02:21:04 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18482#post-id-18482Comment by grtwa for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18499#post-id-18499@StevenPuttemans, "New users must wait 2 days before answering their own question." I will post an answer later. Thank you for reminding.Fri, 09 Aug 2013 04:35:41 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18499#post-id-18499Answer by Michael Burdinov for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?answer=18694#post-id-18694SVD method has very high numerical robustness, but this robustness has its price both in space and in time. If you don't expect a degenerate case you can you much lighter methods that should be able to run on x86. For example:
solve(A.t()*A, A.t()*b, x, DECOMP_CHOLESKY);
Tue, 13 Aug 2013 01:05:22 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?answer=18694#post-id-18694Comment by grtwa for <p>SVD method has very high numerical robustness, but this robustness has its price both in space and in time. If you don't expect a degenerate case you can you much lighter methods that should be able to run on x86. For example:</p>
<pre><code>solve(A.t()*A, A.t()*b, x, DECOMP_CHOLESKY);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18708#post-id-18708Thanks, I need deal with some degenerate case. Before using SVD, I intended to use QR. But DECOMP_QR seems to equate with DECOMP_SVD in OpenCV2.4.5.Tue, 13 Aug 2013 02:27:05 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?comment=18708#post-id-18708Answer by grtwa for <p>I use the function "cv::solve(src1, src2, dst, DECOMP_SVD)" to solve a least-squares problem (i.e., src1 * dst = src2). The size of src1 is 29030 * 7809 and the size of src2 is 29030 * 122. Both types of src1 and src2 are CV_32FC1. OpenCV crashes at the 1370 line of "lapack.cpp" and the code of this line is "buffer.allocate(bufsize)". "buffer" is a "AutoBuffer< uchar >" and "bufsize" is 1151016404. Is this problem caused by out-of-memory? My physical memory is 8G and operating system is 64bit Win7.</p>
<pre><code>Mat A(29030,7809,CV_32FC1);
Mat b(29030,122,CV_32FC1);
for(i=0;i<29030;i++)
{
for(j=0;j<7809;j++)
A.at<float>(i,j)=float(i*7809+j);
for(j=0;j<122;j++)
b.at<float>(i,j)=float(i*122+j);
}
Mat x;
solve(A,b,x,DECOMP_SVD);
</code></pre>
http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?answer=18617#post-id-18617I have solved this problem by convert the compiling platform to x64. On x86 platform, only 2G physical memory can be used.Sun, 11 Aug 2013 23:36:52 -0500http://answers.opencv.org/question/18438/linear-least-squares-for-large-scale-dense-matrix/?answer=18617#post-id-18617