Ask Your Question
1

ArUco marker translation deviation for some video frames

asked 2017-05-08 06:35:59 -0500

eskrs gravatar image

updated 2017-05-08 08:20:28 -0500

We are using ArUco markers for pose estimation and are encountering an issue.

The detected pose translation will sometimes be significantly off without any visible disturbance in the source image.

This is a follow-up to an issue that occurs when corner refinement is turned off. http://answers.opencv.org/question/14... The issue seems to be is less frequent than the one without corner refinement, but it still very desirable to work with the cleanest possible data before any filtering.

Is it possible to fix this issue with ArUco parameters or otherwise?

Below is a test program that takes a file name as argument and two example images to demonstrate the issue. The two images are two consecutive frames of a video and have very little differences.

The program output is:

$ ./arucobug frame104.png 
R: [1.59948, -2.23327, 0.380623]  T: [22.319, -4.29597, 94.5112]
$ ./arucobug frame103.png 
R: [1.5954, -2.22185, 0.386173]  T: [24.8896, -4.84338, 105.443]

So the marker is detected about 10cm (intrinsics are in cm) further away than it should be.

#include "opencv2/opencv.hpp"
#include "opencv2/aruco.hpp"


int main(int argc, char *argv[])
{
    if (argc != 2)
        return 1;


    cv::Mat frame = cv::imread(argv[1]);
    cv::Mat debugFrame = frame.clone();

    // Intrinsic camera parameters.
    cv::Mat camMatrix = cv::Mat::eye(3, 3, CV_64F);
    cv::Mat distortionCoeffs = cv::Mat::zeros(8, 1, CV_64F);
    camMatrix.at<double>(0, 0) = 2.3396381685789738e+03;
    camMatrix.at<double>(0, 2) = 960.;
    camMatrix.at<double>(1, 1) = 2.3396381685789738e+03;
    camMatrix.at<double>(1, 2) = 540.;
    distortionCoeffs.at<double>(0, 0) = -1.0982746232841779e-01;
    distortionCoeffs.at<double>(1, 0) = 2.2689585715220828e-01;
    distortionCoeffs.at<double>(2, 0) = 0.;
    distortionCoeffs.at<double>(3, 0) = 0.;
    distortionCoeffs.at<double>(4, 0) = -2.2112148171171589e-01;

    cv::Ptr<cv::aruco::Dictionary> dict = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_4X4_50);
    cv::Ptr<cv::aruco::DetectorParameters> params = cv::aruco::DetectorParameters::create();
    std::vector<int> markerIds;
    std::vector<std::vector<cv::Point2f>> markerCorners;
    std::vector<cv::Vec3d> rotationVec, translationVec;


    params->cornerRefinementMethod = cv::aruco::CORNER_REFINE_CONTOUR;
    cv::aruco::detectMarkers(frame, dict, markerCorners, markerIds, params);
    auto numMarkers = markerIds.size();

    if (numMarkers > 0)
    {
        cv::aruco::drawDetectedMarkers(debugFrame, markerCorners, markerIds);

        cv::aruco::estimatePoseSingleMarkers(markerCorners, 5.0, camMatrix, distortionCoeffs, rotationVec, translationVec);

        for (int i = 0; i < numMarkers; i++)
        {
            // Official aruco axis drawing.
            cv::aruco::drawAxis(debugFrame, camMatrix, distortionCoeffs, rotationVec[i], translationVec[i], 0.5 * 5.0);

            std::cout << "R: " << rotationVec[i] << "  T: " << translationVec[i] << std::endl;


            // Project marker origin.
            std::vector< cv::Point3f > axisPoints;
            axisPoints.push_back(cv::Point3f(0, 0, 0));
            std::vector< cv::Point2f > imagePoints;
            cv::projectPoints(axisPoints, rotationVec[i], translationVec[i], camMatrix, distortionCoeffs, imagePoints);

            // Convert rotation vector to rotation matrix.
            cv::Mat rotMat;
            cv::Rodrigues(rotationVec[i], rotMat);

            // Check for proper rotation matrix.
            assert(cv::determinant(rotMat) > 0.99 && cv::determinant(rotMat) < 1.01);

            // Manually rotate the z-axis vector.
            cv::Mat testVec(cv::Point3f(0, 0, 10.0f));
            testVec.convertTo(testVec, CV_64FC1);
            cv::Mat testTransformed = rotMat * testVec;

            // Project the rotated z-axis vector.
            std::vector<cv::Point3f> pts;
            pts.push_back ...
(more)
edit retag flag offensive close merge delete

Comments

We haven't got same results :

Corners =[1475.509, 506.11462;
 1428.4612, 402.75797;
 1542.4216, 365.06506;
 1591.9524, 465.51755]
 R = [1.59932, -2.23255, 0.384303]
 T=[22.3176, -4.2958, 94.5001]
 Corners =[1475.4648, 505.03976;
 1428.7175, 401.55618;
 1542.7222, 364.02765;
 1592.1364, 464.73526]
 R = [1.60257, -2.23203, 0.383516]
 T=[22.3142, -4.33533, 94.4549]

Difference could be :

  1. First I have downloaded image from your link it is jpg files and you have png files
  2. my platform is windows with vs 2015
  3. i have compiled opencv with mkl
LBerger gravatar imageLBerger ( 2017-05-08 07:13:42 -0500 )edit
1

Thanks. I didn't know imgur transcodes images. The builtin upload didn't work because the file size was too large. Since I didn't keep the files on disk, I'll have to redo the sample.

eskrs gravatar imageeskrs ( 2017-05-08 08:02:09 -0500 )edit

@LBerger Here is the same scenario https://picload.org/image/rcwrrgww/fr...https://picload.org/image/rcwrrrgr/fr...

Values:

$ ./arucobug frame60.png
id: 0  R: [0.947212, 2.69749, -0.830882]  T: [17.2521, 2.48272, 90.361]
$ ./arucobug frame59.png
id: 0  R: [0.94127, 2.6841, -0.831416]  T: [19.4121, 2.882, 101.687]
eskrs gravatar imageeskrs ( 2017-05-08 08:19:01 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2017-05-08 09:13:58 -0500

LBerger gravatar image

updated 2017-05-08 11:00:27 -0500

I think you ask too much precision :

float markerLength=5;
std::vector<cv::Vec3f> ptMarker={ cv::Vec3f(-markerLength / 2.f, markerLength / 2.f, 0),cv::Vec3f(markerLength / 2.f, markerLength / 2.f, 0),
    cv::Vec3f(markerLength / 2.f, -markerLength / 2.f, 0),cv::Vec3f(-markerLength / 2.f, -markerLength / 2.f, 0) };

params->cornerRefinementMethod = cv::aruco::CORNER_REFINE_CONTOUR;
params->adaptiveThreshConstant=true;
cv::aruco::detectMarkers(frame, dict, markerCorners, markerIds, params);
auto numMarkers = markerIds.size();

if (numMarkers > 0)
{
    cv::aruco::drawDetectedMarkers(debugFrame, markerCorners, markerIds);

    cv::aruco::estimatePoseSingleMarkers(markerCorners, markerLength, camMatrix, distortionCoeffs, rotationVec, translationVec);

    for (int i = 0; i < numMarkers; i++)
    {
        // Official aruco axis drawing.
        cv::Mat imagePt;
        cv::aruco::drawAxis(debugFrame, camMatrix, distortionCoeffs, rotationVec[i], translationVec[i], 0.5 * 5.0);
        std::cout << "\n Corners =" << markerCorners[i];
        cv::projectPoints(ptMarker, rotationVec, translationVec, camMatrix, distortionCoeffs,imagePt);
        std::cout<<"\n ImagePt = "<<imagePt;
        std::cout << "\n R = "<<rotationVec[i] <<"\n T="<< translationVec[i];
        for (int j=0;j<ptMarker.size();j++)
            std::cout<<"\n"<<cv::norm(imagePt.at<cv::Point2f>(j,0)-markerCorners[i][j]);

and results are :

 Corners =[1496.6306, 612.09216;
 1403.8015, 682.35718;
 1313.5145, 596.81036;
 1406.2969, 528.19458]
 ImagePt = [1496.7004, 611.69116;
 1403.7068, 682.76807;
 1313.6329, 596.41217;
 1406.2034, 528.58215]
 R = [0.948757, 2.69793, -0.828673]
 T=[17.2317, 2.47978, 90.2495]
0.407035
0.421666
0.415426
0.398693
 Corners =[1496.6405, 614.14752;
 1403.9784, 684.52588;
 1313.4429, 598.77551;
 1406.4125, 530.15881]
 ImagePt = [1496.7953, 613.77832;
 1403.8063, 684.91089;
 1313.6472, 598.3988;
 1406.2253, 530.51898]
 R = [0.948401, 2.69694, -0.827097]
 T=[17.223, 2.55703, 90.191]
0.400335
0.421732
0.428564
0.405882

corners and imagePt distance is less than a pixel

edit flag offensive delete link more

Comments

Thank you. I don't think I can follow your reasoning here. You are showing that the returned transformation maps the image and world coordinates correctly. This is expected since ArUco uses the corners to determine the transformation. What I am questioning is the corner positions in the first place. They are significantly far (10px+) inside the actual marker corners.

eskrs gravatar imageeskrs ( 2017-05-08 10:26:55 -0500 )edit

Sorry I missed this problem : add params->adaptiveThreshConstant=true;

LBerger gravatar imageLBerger ( 2017-05-08 11:01:27 -0500 )edit

Question Tools

1 follower

Stats

Asked: 2017-05-08 06:35:59 -0500

Seen: 766 times

Last updated: May 08 '17