lovaj's profile - activity

This is my first image processing application, so please be kind with this filthy peasant.

THE APPLICATION:

I want to implement a fast application (performance are crucial even over accuracy) where given a photo (taken by mobile phone) containing a movie poster finds the most similar photo in a given dataset and return a similarity score. The dataset is composed by similar pictures (taken by mobile phone, containing a movie poster). The images can be of different size, resolutions and can be taken from different viewpoints (but there is no rotation, since the posters are supposed to always be right-oriented).

Any suggestion on how to implement such an application is well accepted.

FEATURE DESCRIPTIONS IN OPENCV:

I've never used OpenCV and I've read this tutorial about Feature Detection and Description by OpenCV.

From what I've understood, these algorithms are supposed to find keypoints (usually corners) and eventually define descriptors (which describe each keypoint and are used for matching two different images). I used "eventually" since some of them (eg FAST) provides only keypoints.

MOST SIMILAR IMAGE PROBLEM AND LSH:

The problems above doesn't solve the problem "given an image, how to find the most similar one in a dataset in a fast way". In order to do that, we can both use the keypoints and descriptors obtained by any of the previous algorithms. The problem stated above seems like a nearest neighbor problem and Locality Sensitive Hashing is a fast and popular solution for find an approximate solution for this problem in high-dimensionality spaces.

THE QUESTION:

What I don't understand is how to use the result of any of the previous algorithms (i.e. keypoints and descriptors) in LSH.

Is there any implementation that treat this problem?

I want to build OpenCV 3.2 for AVX-2 support. By using -DENABLE_AVX2=ON and using gcc I get these compiler flags:

-msse -msse2 -mno-avx -mavx2

But following this I should use -DCPU_BASELINE=AVX2, but then I get these compiler flags:

-msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2

What is the correct way to do this?

In addition, I have a AVX-512 compatible machine (Intel KNL) and I would obtain even better performances if I could build it using these set of instructions. I've seen here some trace of 512 stuff, but I can't understand how could I use it.

Thanks @Tetragramm I think you did the best you could, I really appreaciate it and your help was crucial in my project! Chapeau sir! :)

@Tetragramm thanks for your answer, you are the one who went closer to an actual solution between all the people that I've talked to! However (as you can see in my updated answer), the result is kinda approximated, i.e. some values are different...For example, you can see how in img some values are different from myImg obtained through your approach. I don't know if there is a solution for this. Btw, I would need the boolean value, could you please explain a little better the last part?

@Tetragram I've opened this question about the input and output images that you asked, please help!

I want to optimize this code, in particular this function:

bool interpolate(const Mat &im, float ofsx, float ofsy, float a11, float a12, float a21, float a22, Mat &res)
{         
   bool ret = false;
   // input size (-1 for the safe bilinear interpolation)
   const int width = im.cols-1;
   const int height = im.rows-1;
   // output size
   const int halfWidth  = res.cols >> 1;
   const int halfHeight = res.rows >> 1;
   int dim = res.rows * res.cols;
   float *out = res.ptr<float>(0);
   for (int j=-halfHeight; j<=halfHeight; ++j)
   {
      const float rx = ofsx + j * a12;
      const float ry = ofsy + j * a22;
      #pragma omp simd
      for(int i=-halfWidth; i<=halfWidth; ++i)
      {
         float wx = rx + i * a11;
         float wy = ry + i * a21;
         const int x = (int) floor(wx);
         const int y = (int) floor(wy);
         if (x >= 0 && y >= 0 && x < width && y < height)
         {
            // compute weights
            wx -= x; wy -= y;
            // bilinear interpolation
            *out++ = 
               (1.0f - wy) * ((1.0f - wx) * im.at<float>(y,x)   + wx * im.at<float>(y,x+1)) +
               (       wy) * ((1.0f - wx) * im.at<float>(y+1,x) + wx * im.at<float>(y+1,x+1));
         } else {
            *out++ = 0;
            ret =  true; // touching boundary of the input            
         }
      }
   }
   return ret;
}

Can anybody please help to find an equivalent function in OpenCV for the code above? I'm not expert on image processing, so I don't really know what the fucntion above does in details, but I think this is a warp-affine transformation, even though I don't really know how to define an OpenCV equivalent.

It seems that the input images are the blurred image, the original image or small patches of it. The output result is always a small patch.

Here are some samples:

Sample 1:

Args:

ofx=175.497 ofsy=315.06 a11=1.69477 a12=0.0671724 a21=0.0679493 a22=1.56309

Input image:

Output image:

Sample 2:

Args:

ofx=572.121 ofsy=326.659 a11=0.871508 a12=0 a21=0.346405 a22=1.14744

Input image:

Output image:

Sample 3:

ofx=66.571 ofsy=148.991 a11=1.12027 a12=0.126609 a21=0.126609 a22=2.53436

Input image:

Output image:

These are different sections of the original code where the function is called:

  // warp input according to current shape matrix
  interpolate(wrapper.prevBlur, lx, ly, u11*ratio, u12*ratio, u21*ratio, u22*ratio, img);

  Mat smoothed(patchImageSize, patchImageSize, CV_32FC1, (void *)&workspace.front());
  // interpolate with det == 1
  if (!interpolate(img, x, y, a11, a12, a21, a22, smoothed))
  {

  // subsample with corresponding scale
  interpolate(smoothed, (float)(patchImageSize>>1), (float)(patchImageSize>>1), imageToPatchScale, 0, 0, imageToPatchScale, patch);


  // ok, do the interpolation
  interpolate(img, x, y, a11, a12, a21, a22, patch);

@Tetragramm I'll post the images very soon as you asked btw ;)

@Tetragramm thanks again for your help. I'll try to do In this question you can see a little more details.

@Tetragramm thanks for you help. I kinda understand the process, but I'm a little bit confused by how to use the code...and you know, just a little mistake and you get 0 results in image processing! Could you PLEASE help me writing the correct code?

@pi-null-mezon I'm sorry, it was already optimized on that point, I updated the question. Can you help me in some other way? I'm seriously stucked on this, using an equivalent function could help me so much

I have this interpolate function taken from this code which I want to optimize:

bool interpolate(const Mat &im, float ofsx, float ofsy, float a11, float a12, float a21, float a22, Mat &res)
{         
   bool ret = false;
   // input size (-1 for the safe bilinear interpolation)
   const int width = im.cols-1;
   const int height = im.rows-1;
   // output size
   const int halfWidth  = res.cols >> 1;
   const int halfHeight = res.rows >> 1;
   int dim = res.rows * res.cols;
   float *out = res.ptr<float>(0);
   const float *imptr  = im.ptr<float>(0);
   for (int j=-halfHeight; j<=halfHeight; ++j)
   {
      const float rx = ofsx + j * a12;
      const float ry = ofsy + j * a22;
      #pragma omp simd
      for(int i=-halfWidth; i<=halfWidth; ++i)
      {
         float wx = rx + i * a11;
         float wy = ry + i * a21;
         const int x = (int) floor(wx);
         const int y = (int) floor(wy);
         if (x >= 0 && y >= 0 && x < width && y < height)
         {
            // compute weights
            wx -= x; wy -= y;
            // bilinear interpolation
            *out++ = 
            (1.0f - wy) * ((1.0f - wx) * imptr[rowOffset+x]   + wx * imptr[rowOffset+x+1]) +
            (       wy) * ((1.0f - wx) * imptr[rowOffset1+x] + wx * imptr[rowOffset1+x+1]);
        } else {
            *out++ = 0;
            ret =  true; // touching boundary of the input            
         }
      }
   }
   return ret;
}

I'm not the image processing guy and I'm really struggling to perform the equivalent using the opencv implementation. This is crucial for my project since the function above is incredibly time consuming.

Please help.

Disclamer: I'm not a cmake expert, I used it only to build OpenCV.

Reading this it seems that we need to specify that we want to build OpenCV with MKL and simd support.

So do I have to add the following flags?

-DENABLE_AVX2=ON -DHAVE_MKL=ON -DMKL_WITH_TBB=ON

Digging a little bit on opencv repository in this file it seems we need only to enable ipp (with -DWITH_IPP=ON and optionally specify -DWITH_IPP if we want to provide our version of IPP).

Am I correct or I'm missing something?

But this is going to copy element by element, right? It's not just copying the references, right? In other words: the cost is linear, right?

@matman if you would give a look at this question I would really appreciate it

@matman speaking of which, how do I build OpenCV with AVX support?

I've this function:

void foo(cv::Mat &mat){
  float *p = mat.ptr<float>(0);
  //modify mat values through p
}

And I have this code that calls the function:

void bar(std::vector<unsigned char> &vec){
  //...
  cv::Mat res(m, n, CV_32FC1, (void *)&workspace.front());
}

However, the code above has a performance problem: vec isn't probably aligned. In fact, the Intel compiler says that reference *out has unaligned access. I hope that this is the reason.

Btw, as I found out, cv::Mat is going to be aligned. So a simple workaround would be to:

1. Create cv::Mat res(m,n)
2. Call foo(m);
3. Assign the vec pointer to m.ptr<float>(0)

As you can imagine, performance here are crucial, so deep-copies are out of questions.

I tried the code in this question:

vec = res.data;

But I get a compiler error. Besides, I don't know if this code above is going to do an (inefficient) copy, or just change the pointed data by vec.

How can I do this smoothly? Otherwise, another workaround would be appreciated.

Disclamer: I am a SIMD newbie, but I have a good knowledge of OpenCV.

Let's suppose I have basic OpenCV code.

cv::Mat1f a = cv::Mat(n, n);
cv::Mat1f b = cv::Mat(n, n);
cv::Mat1f x;
//fill a and b somehow
x = a+b;

Now, let's suppose I want to use the code above on AVX2 or even AVX-512 machine. Having the data 32-aligned could be a great benefit for performances. The data above should not be aligned. I'm almost certain of it because in the optimization report files generated by the Intel Compiler it's said that the data is unaligned, but I could be wrong.

So what if I allocate 32-aligned pointers and use them for the operations above? Something like:

float *apt = __mm_alloc(sizeof(float)*m*n, 32);
float *bpt = __mm_alloc(sizeof(float)*m*n, 32);
float *xpt = __mm_alloc(sizeof(float)*m*n, 32);
cv::Mat1f a = cv::Mat(n, m, apt);
cv::Mat1f b = cv::Mat(m, n, bpt);
cv::Mat1f x = cv::Mat(n, n, xpt);
x = a+b;

I see three problems that concerns me in the code above:

1. Matrix multiplications in cv::Mat returns a new cv::Mat object which, unfortunately, would be not aligned.
2. Am I reinventing the wheel? I don't know if OpenCV includes already this kind of optimizations, even though I didn't find nothing useful on Google.
3. Reading a little bit on Intel Intrinsics, when we use __mm_malloc, we should use __mm_free. However, cv::Mat are "kinda" of smart pointers, so they're allocated and de-allocated behind-the-hood, where probably some free or delete function is called.

To overcome problem 1, in very critical sections of code where data alignment could improve the performance, I could get the pointer back and do the sum operation using pointers. Something like:

//do the code above
float *aptr = a.ptr<float>(0);
float *bptr = b.ptr<float>(0);
float *xptr = c.ptr<float>(0);
for(int i=0; i<n ; i++)
  for(int j=0; j<m; j++)
    xptr[i*m+j] = a[i*m+j] + b[i*m+j];

But again, I'm afraid that I'm reinventing the wheel. Notice that this is a toy example to make it MCV, my actual code is much more complicated than this and compiler optimizations may not be obvious.

I compile my code with icpc and the following flags:

INTEL_OPT=-O3 -ipo -simd -xCORE-AVX2 -parallel -qopenmp -fargument-noalias -ansi-alias -no-prec-div -fp-model fast=2 -fma -align -finline-functions
INTEL_PROFILE=-g -qopt-report=5 -Bdynamic -shared-intel -debug inline-debug-info -qopenmp-link dynamic -parallel-source-info=2 -ldl

@LBerger daaaaaamn in the whole project width=im.cols, this was an hell of a joke! Thanks so much for noticing it!

I have this function:

bool interpolate(const Mat &im, float ofsx, float ofsy, float a11, float a12, float a21, float a22, Mat &res)
{         
   // input size (-1 for the safe bilinear interpolation)
   const int width = im.cols-1;
   const int height = im.rows-1;
   // output size
   const int halfWidth  = res.cols >> 1;
   const int halfHeight = res.rows >> 1;
   float *out = res.ptr<float>(0);
   const float *imptr  = im.ptr<float>(0);
   const float *resptr = im.ptr<float>(0);
   for (int j=-halfHeight; j<=halfHeight; ++j)
   {
      //...
      for(int i=-halfWidth; i<=halfWidth; ++i, out++)
      {
         const int x = (int) //something;
         const int y = (int) //something;
         if (x >= 0 && y >= 0 && x < width && y < height)
         {
            std::cout<<"im(y,x)="<<im.at<float>(y,x)<<" imptr[y*width+x]="<<imptr[y*width+x]<<std::endl;    
         }
      }
   }
   return ret;
}

Where I want to use const float *imptr instead of const cv::Mat img for better efficiency. However, I can't understand why, but imptr[y*width+x]!=im.at<float>(y,x). Why?

I built OpenCV using IPP, but I want to be sure that cv::GuassianBlur is executed by using that version and not other implementations.

I'm having this doubt because I've built OpenCV with OpenCL turned on, and I see on the stack call a lot using the libnvidia-opencl, while I never use OpenCL/CUDA/whatever-GPU-stuff in my code.

I'll rebuild OpenCV with -DWITH_OPENCL=OFF, but I want to be sure somehow that I use the IPP implementation. This is because I've installed the latest version of IPP and I can use very performant Intel CPU.

@berak that was what I was thinking about!

@berak is it correct to say that this is going to be very inefficient in terms of performance because of the deep copy? This is for an HPC application, I need to do this efficiently as possible

@Tetragramm Btw I don't really understand your point about the difference and the 255 scale. I read the image, they have value range 0-255, which is normal. I use the gaussianBlur, and the result is in 0-255. The diff is too in 0-255. Then, I display the image in this 0-255 system. What is the problem in all this? The convertTo(imageF_8UC3, CV_8U, 255); is used only when I use imwrite, not when I use imshow. So to see the true diff you propose to use diff /= 255 instead of diff, right?

BUT if the difference was so small, my algorithm would generate almost the same number of keypoints, right? Instead using the code with chain of blurs (the correct, original version) generates 1760 keypoints, instead with one blur 2397! Something's is wrong here and I'm really stucked trying to understand what XD

@Tetragramm from my understanding of your answer, you say that the blurs obtained from the chain version and the ones obtained with one shot are the same...But then why the number of produced keypoints is different? The only explanation is that the produced blurs are different.

@Tetragramm I'm sorry, I'm missing your point. By "your problem" you mean how images are shown/saved, or why the single blur formula doesn't work? I'm sorry if I didn't understand your answer, if you can give me more details it would be much appreciated