Revision history [back]

Hi axadiw,

I had the same problem and all the samples i found on the net were crappy hacks!

The 3 most important things to know are that :

• solvePnP gives you the transfer matrix from the model's frame (ie the cube) to the camera's frame (it's called view matrix).
• The camera's frame are not the same in opencv and opengl. Y axis and Z axis are inverted.
• How matrixes are stored is not the same neither. Opengl matrixes are column major order whereas they are row major order in Opencv.

So to compute the view matrix (transfer matrix from the model's frame to the camera's frame) that will be used in OpenGl, you have to:

• Use same coordinates to draw the cube in Opengl and to compute the camera's pose with solvePnP (markerObjectPoints)

• build the view matrix like this:

  cv::Mat rvec, tvec;
  cv::solvePnP(objectPoints, imagePoints, intrinsics, distortion, rvec, tvec, ...);
  cv::Mat rotation, viewMatrix(4, 4, CV_64F);
  cv::Rodrigues(rvec, rotation);
  
  for(unsigned int row=0; row<3; ++row)
  {
     for(unsigned int col=0; col<3; ++col)
     {
        viewMatrix.at<double>(row, col) = rotation.at<double>(row, col);
     }
     viewMatrix.at<double>(row, 3) = tvec.at<double>(row, 0);
  }
  viewMatrix.at<double>(row, 3) = 1.0f;
  

• Multiply the view matrix by the transfer matrix between OpenCV and OpenGL:

  cv::Mat cvToGl = cv::Mat::zeros(4, 4, CV_64F); cvToGl.at<double>(0, 0) = 1.0f; cvToGl.at<double>(1, 1) = -1.0f; // Invert the y axis cvToGl.at<double>(2, 2) = -1.0f; // invert the x axis cvToGl.at<double>(3, 3) = 1.0f; viewMatrix = cvToGl * viewMatrix;

• Because OpenCV's matrixes are stored by row you have to transpose the matrix in order that OpenGL can read it by column:

  cv::Mat glViewMatrix = cv::Mat::zeros(4, 4, CV_64F);
  cv::transpose(viewMatrix , glViewMatrix);
  glMatrixMode(GL_MODELVIEW);
  glLoadMatrixd(&glViewMatrix.at<double>(0, 0));
  

  And after that it should work fine ;) Moreover by watching your video, i can notice a shift between the cube and the marker, so i think you probably have calibration problems. Try with default values to see if it's better..

I hope it will be useful ;)

Hi axadiw,

I had the same problem and all the samples i found on the net were crappy hacks!

The 3 most important things to know are that :

• solvePnP gives you the transfer matrix from the model's frame (ie the cube) to the camera's frame (it's called view matrix).
• The camera's frame are not the same in opencv and opengl. Y axis and Z axis are inverted.
• How matrixes are stored is not the same neither. Opengl matrixes are column major order whereas they are row major order in Opencv.

So to compute the view matrix (transfer matrix from the model's frame to the camera's frame) that will be used in OpenGl, you have to:

• Use same coordinates to draw the cube in Opengl and to compute the camera's pose with solvePnP (markerObjectPoints)

• build the view matrix like this:

  cv::Mat rvec, tvec;
  cv::solvePnP(objectPoints, imagePoints, intrinsics, distortion, rvec, tvec, ...);
  cv::Mat rotation, viewMatrix(4, 4, CV_64F);
  cv::Rodrigues(rvec, rotation);
  
  for(unsigned int row=0; row<3; ++row)
  {
     for(unsigned int col=0; col<3; ++col)
     {
        viewMatrix.at<double>(row, col) = rotation.at<double>(row, col);
     }
     viewMatrix.at<double>(row, 3) = tvec.at<double>(row, 0);
  }
  viewMatrix.at<double>(row, viewMatrix.at<double>(3, 3) = 1.0f;
  

• Multiply the view matrix by the transfer matrix between OpenCV and OpenGL:

  cv::Mat cvToGl = cv::Mat::zeros(4, 4, CV_64F); cvToGl.at<double>(0, 0) = 1.0f; cvToGl.at<double>(1, 1) = -1.0f; // Invert the y axis cvToGl.at<double>(2, 2) = -1.0f; // invert the x axis cvToGl.at<double>(3, 3) = 1.0f; viewMatrix = cvToGl * viewMatrix;

• Because OpenCV's matrixes are stored by row you have to transpose the matrix in order that OpenGL can read it by column:

  cv::Mat glViewMatrix = cv::Mat::zeros(4, 4, CV_64F);
  cv::transpose(viewMatrix , glViewMatrix);
  glMatrixMode(GL_MODELVIEW);
  glLoadMatrixd(&glViewMatrix.at<double>(0, 0));
  

  And after that it should work fine ;) Moreover by watching your video, i can notice a shift between the cube and the marker, so i think you probably have calibration problems. Try with default values to see if it's better..

I hope it will be useful ;)

Hi axadiw,

I had the same problem and all the samples i found on the net were crappy hacks!

The 3 most important things to know are that :

• solvePnP gives you the transfer matrix from the model's frame (ie the cube) to the camera's frame (it's called view matrix).
• The camera's frame are not the same in opencv and opengl. Y axis and Z axis are inverted.
• How matrixes are stored is not the same neither. Opengl matrixes are column major order whereas they are row major order in Opencv.

So to compute the view matrix (transfer matrix from the model's frame to the camera's frame) that will be used in OpenGl, you have to:

• Use same coordinates to draw the cube in Opengl and to compute the camera's pose with solvePnP (markerObjectPoints)

• build the view matrix like this:

  cv::Mat rvec, tvec;
  cv::solvePnP(objectPoints, imagePoints, intrinsics, distortion, rvec, tvec, ...);
  cv::Mat rotation, viewMatrix(4, 4, CV_64F);
  cv::Rodrigues(rvec, rotation);
  
  for(unsigned int row=0; row<3; ++row)
  {
     for(unsigned int col=0; col<3; ++col)
     {
        viewMatrix.at<double>(row, col) = rotation.at<double>(row, col);
     }
     viewMatrix.at<double>(row, 3) = tvec.at<double>(row, 0);
  }
  viewMatrix.at<double>(3, 3) = 1.0f;
  

• Multiply the view matrix by the transfer matrix between OpenCV and OpenGL:

  cv::Mat cvToGl = cv::Mat::zeros(4, 4, CV_64F); cvToGl.at<double>(0, 0) = 1.0f; cvToGl.at<double>(1, 1) = -1.0f; // Invert the y axis cvToGl.at<double>(2, 2) = -1.0f; // invert the x z axis cvToGl.at<double>(3, 3) = 1.0f; viewMatrix = cvToGl * viewMatrix;

• Because OpenCV's matrixes are stored by row you have to transpose the matrix in order that OpenGL can read it by column:

  cv::Mat glViewMatrix = cv::Mat::zeros(4, 4, CV_64F);
  cv::transpose(viewMatrix , glViewMatrix);
  glMatrixMode(GL_MODELVIEW);
  glLoadMatrixd(&glViewMatrix.at<double>(0, 0));
  

  And after that it should work fine ;) Moreover by watching your video, i can notice a shift between the cube and the marker, so i think you probably have calibration problems. Try with default values to see if it's better..

I hope it will be useful ;)

Hi axadiw,

I had the same problem and all the samples i found on the net were crappy hacks!

The 3 most important things to know are that :

• solvePnP gives you the transfer matrix from the model's frame (ie the cube) to the camera's frame (it's called view matrix).
• The camera's frame are not the same in opencv and opengl. Y axis and Z axis are inverted.
• How matrixes are stored is not the same neither. Opengl matrixes are column major order whereas they are row major order in Opencv.

So to compute the view matrix (transfer matrix from the model's frame to the camera's frame) that will be used in OpenGl, you have to:

• Use same coordinates to draw the cube in Opengl and to compute the camera's pose with solvePnP (markerObjectPoints)

• build the view matrix like this:

  cv::Mat rvec, tvec;
  cv::solvePnP(objectPoints, imagePoints, intrinsics, distortion, rvec, tvec, ...);
  cv::Mat rotation, viewMatrix(4, 4, CV_64F);
  cv::Rodrigues(rvec, rotation);
  
  for(unsigned int row=0; row<3; ++row)
  {
     for(unsigned int col=0; col<3; ++col)
     {
        viewMatrix.at<double>(row, col) = rotation.at<double>(row, col);
     }
     viewMatrix.at<double>(row, 3) = tvec.at<double>(row, 0);
  }
  viewMatrix.at<double>(3, 3) = 1.0f;
  

• Multiply the view matrix by the transfer matrix between OpenCV and OpenGL:

  cv::Mat cvToGl = cv::Mat::zeros(4, 4, CV_64F);
   cvToGl.at<double>(0, 0) = 1.0f;
   cvToGl.at<double>(1, 1) = -1.0f; // Invert the y axis
   cvToGl.at<double>(2, 2) = -1.0f; // invert the z axis
   cvToGl.at<double>(3, 3) = 1.0f;
   viewMatrix = cvToGl * viewMatrix;
  viewMatrix;
  

• Because OpenCV's matrixes are stored by row you have to transpose the matrix in order that OpenGL can read it by column:

  cv::Mat glViewMatrix = cv::Mat::zeros(4, 4, CV_64F);
  cv::transpose(viewMatrix , glViewMatrix);
  glMatrixMode(GL_MODELVIEW);
  glLoadMatrixd(&glViewMatrix.at<double>(0, 0));
  

  And after that it should work fine ;) Moreover by watching your video, i can notice a shift between the cube and the marker, so i think you probably have calibration problems. Try with default values to see if it's better..

I hope it will be useful ;)