OpenCV Q&A Forum - RSS feedhttp://answers.opencv.org/questions/OpenCV answersenCopyright <a href="http://www.opencv.org">OpenCV foundation</a>, 2012-2018.Tue, 01 Dec 2020 10:18:57 -0600Pose estimation of image from a point cloudhttp://answers.opencv.org/question/238529/pose-estimation-of-image-from-a-point-cloud/I have a 3D point cloud of scene, its set of views with known poses and a new query image from camera with camera calibration data (i.e. focal length). Here is a snapshot of my point cloud of scene
![image description](/upfiles/16068386634272687.png)
and query image
![image description](/upfiles/16068386779254414.jpg)
I need to estimate the camera pose of the query image.
I have found similar [question](https://stackoverflow.com/questions/44445851/how-to-estimate-camera-pose-matrix-of-a-new-image-i-from-a-known-3d-point-cloud) on SO. My intuition is to find point correspondences btw the query image and all the view image of the scene and apply SolvePnP to get the camera pose.
My questions is whether it is right solution and how to approach it? Should I somehow use the original point cloud here once I get good correspondences btw the query image and view images to estimate the camera pose?sigmoid90Tue, 01 Dec 2020 10:18:57 -0600http://answers.opencv.org/question/238529/Projecting point cloud to image plane in order to estimate camera posehttp://answers.opencv.org/question/238349/projecting-point-cloud-to-image-plane-in-order-to-estimate-camera-pose/I am working on a task of image registration in point cloud of scene. I have an image from camera, 3D point cloud of scene and camera calibration data (i.e. focal length). The image and point cloud of scene share the same space.
Here is a snapshot of my point cloud of scene
![image description](/upfiles/16067296657068877.png)
and query image
![image description](/upfiles/16067296871448465.jpg)
From my previous question [here](https://answers.opencv.org/question/238269/error-assertion-failed-scn-3-scn-4-in-cvtcolor-when-calculating-orb-features-on-point-cloud/) I have learnt that the SolvePnP method from OpenCV would work in this case but there is no obvious way to find robust correspondences btw query image and scene point cloud.
My intuition now is to project point cloud to image plane, match keypoints calculated on it against ones calculated on query image and use robust point correspondences in the SolvePnP method to get camera pose.
I have found [here](https://stackoverflow.com/questions/40677116/conversion-of-cloud-data-into-2d-image-using-opencv) a way to project point cloud to image plane.
My questions is could it work and is it possible to preserve the transformation btw the resultant image of scene and the original point cloud?sigmoid90Fri, 27 Nov 2020 05:36:50 -0600http://answers.opencv.org/question/238349/why invert posehttp://answers.opencv.org/question/232411/why-invert-pose/ I have a pose output from solvepnp(), not a prob. Many sources suggest to take the inverse of the pose. I am having a contextual blindspot here: can someone tell me why I should use the inverted pose. In common sense language how is the inverted coordinate space more accurate/more appropriate.superflySun, 12 Jul 2020 13:15:56 -0500http://answers.opencv.org/question/232411/How to get 2d correspondances for 3d cad model points for non textured objects?http://answers.opencv.org/question/231877/how-to-get-2d-correspondances-for-3d-cad-model-points-for-non-textured-objects/Hi
I have to estimate a very accurate 6D Pose of regular shape objects (like blocks for construction ) without any textured. I have the 3D CAD model of the object, and I know roughly the coordinates of the blocks. So the camera is pointing down to the object and can use stereo cameras or ZED
I know roughly the coordinates on the object and the camera will be placed around 1m away from the object. I would like to use OpenCV `solvePnP` method and for that need to find the 2d correspondences of the scene image for 3D CAD model points of the objects.
I search and found this tutorial for [Real Time pose estimation of a textured object](https://docs.opencv.org/master/dc/d2c/tutorial_real_time_pose.html) but in my case the objects are nontextured. So I can not use such key descriptors like SURF or SIFT or ORB. And also I would like to use it in real-time and as input to take my camera input.
Hence, research in this area has focused on tracking using natural features, such as edge-based, optical flow-based,template-based, and keypoint-based. Is this relevant so I can find the 2d correspondences for 3d cad model points for non-textured objects like construction blocks and can get the 6D Pose?
Also, I know that exists Harris Corners, Hough Lines, Canny Edge, or Fast Corners descriptors but not sure if using them can track the object and find the 2D-3D points correspondences. Can I use them in [ Real Time pose estimation of a textured object](https://docs.opencv.org/master/dc/d2c/tutorial_real_time_pose.html) and modify the source code for real-time so I can track my non-textured objects? If yes what should I change in the source code and how to run it in real-time?
Any help would appreciate.
astronautTue, 30 Jun 2020 09:44:48 -0500http://answers.opencv.org/question/231877/How to get the 6D Pose knowing the 3D CAD model and 2DImage of the object?http://answers.opencv.org/question/231598/how-to-get-the-6d-pose-knowing-the-3d-cad-model-and-2dimage-of-the-object/Hi
I have the 3D CAD Model of the Object Its a simple Box without texture. The image of the object is like this:
![Box](/upfiles/1593008925584958.jpg)
I would like to get the 6D Pose of the box while the camera is swinging. I know can use [solvePnP](https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html?highlight=solvepnp#solvepnp) but for my application, the object is in a fixed position the scene and the camera is swinging(moving). Would be possible to track the object and get the 6D pose in real-time? what is the update rate of the algorithm? So, the 3D CAD model must be in YAML format or any other format works?
Any code I can start with? Also, can I get the error of the position and the orientation, I mean the difference between the ground truth and the estimated 6D pose?
ThanksastronautWed, 24 Jun 2020 09:36:17 -0500http://answers.opencv.org/question/231598/How to properly estimate the Z rotation using ArUco Markershttp://answers.opencv.org/question/228612/how-to-properly-estimate-the-z-rotation-using-aruco-markers/ ![image description](/upfiles/15861938376124694.png)
I am trying to determine the car's (x,y) coordinates and its rotation angle theta, illustrated above. (Picture represents a very simple car model as seen from above)
I will attach my phone on top of the car, and using ArUco markers I would like to determine my car's/phone's pose relative to the world, so that I can extract the X and Y transalations, and the Z rotation which should match theta.
I have managed to shift from camera coordinate system to marker coordinate system using the answer to this question: [Pose of camera from the pose of marker](https://stackoverflow.com/questions/51476702/pose-of-camera-from-the-pose-of-marker)
Here is a result:
![image description](/upfiles/15861946518666403.png)
The 'new rVec' and 'new tVec' represent the rotation and translation vector in the marker's coordinate system.
In this picture everything is fine. My rotation vector is good as I am currently holding my phone at a >90 degree angle (1.57 rad) on X and the translation vector matches the distances.
I can also see by using the drawAxis method provided in the ArUco library that the marker is detected correctly, so I expected the results to be good.
Here however:
![image description](/upfiles/15861947535431555.png)
I have rotated my phone 90 degrees toward the left (on the Z axis) and moved the phone to the right so that I can capture the marker in the picture. The translation vector and the drawAxis give me good results, but the rotation vector is not what I was expecting. I was expecting the rotation on Z to match my theta angle, which should have been around 1.57 the equivalent of pi/2. I also have some rotation on both X and Y which was unexpected, as I thought I should only have a >90 degree angle on Y, and a 0 degree angle on X (The opposite of the previous picture).
I have also noticed some other weird behaviour. Here is a video I made that shows my problems:
[ArUco rotation Troubleshoot](https://www.youtube.com/watch?v=gQzHTcGNAzU&feature=youtu.be)
There are 2 problems I see:
1) The marker is rotated at a 180 degrees angle yet the z rotation vector isn't close to 3.14 (pi).
2) I am intentionally moving the camera, rotating it on the X axis. For some reason this movement modifies the Z and Y rotation as well.
My main idea was to get the rotation of the camera relative to the marker's coordinate system, and extract the Z rotation. Shouldn't the rotation on Z match the theta angle provided in the first picture? I can also provide the code for the pose estimation but I didn't think it's necessary since the drawAxis method's results seem perfect.
Also, if there are any easier or better ways to determine the theta angle using a camera attached to a car please let me know!
stefanos.dimosMon, 06 Apr 2020 13:06:20 -0500http://answers.opencv.org/question/228612/Pose estimation of a small cube shape markerhttp://answers.opencv.org/question/224281/pose-estimation-of-a-small-cube-shape-marker/Hi,
I would like to estimate the pose of a needle, looks like a thin cylinder. I think it is hard to directly estimate the pose of the needle. So I want to attach a small cubic marker on one end of the needle. Assume I know the transform between the needle and the cubic marker, to find the pose the needle, I just need to estimate the pose of the cubic marker. Of course, if anybody can give me suggestions about directly estimating the pose of the needle without a marker, I would really appreciate!
To estimate the pose of the cubic marker, I can put different image on each side of the cube. This method looks like multi-targets of Vuforia, but I want to implement it from scratch.
I think the steps looks like the follows:
calculate features of each image offline (totally six images)
then, for each frame,
1. detect the bounding box of the small cubic marker on the image
2. online calculate the features such as SIFT inside the detected bounding box
3. match the online feature with the offline features to find correspondence
4. calculate homography then the relative pose
I am not sure if the above steps are OK.
Any suggestions are really appreciated.
Thanks a lot.
YLARYL518Thu, 02 Jan 2020 04:21:05 -0600http://answers.opencv.org/question/224281/real_time_pose_estimation VS-c++ compiling issueshttp://answers.opencv.org/question/220859/real_time_pose_estimation-vs-c-compiling-issues/Dear Opencv'rs
I have been able to compile and execute every sample I have touched from the opencv 4.1.1 samples folder, including camera_calibration.
Now, with real_time_pose_etimation I have been facing a sturbon compiling issue -unsuccesfully troubleshooted it for days- so I must kindly ask for your help.
Opencv 4.1.1
Windows 10 Pro, have tried building both x86 and x64
VS 2019, ver 16.3.5
Code: C:\opencv\sources\samples\cpp\tutorial_code\calib3d\real_time_pose_estimation\src
What I usually do -not a cmaker user- is I CTRL+V into a new project source file the main cpp content and copy all the reamining .cpp and .h files to the include/opencv2.
In the case of real_time_pose_estimation I brought a copy of main_detection.cpp as my project main source. Everything else I've dumped into my local "..include/opencv2". I have copied the data folder to my project forlder. This is the way you -I succesfully have- build opencv modules (apps) in VS-c++.
All my include lines seem to be okay, as well as the access to files (as per the path environment var and project configurations require).
When I try to compile, I am getting 36 warnings -27 errors- most of which refer to type operator overflowing after regular math operations. There are many
cpp libraries throwing the warnings, but just one line -which I believe and hoped- from the main_detection.cpp line 351 (in my VS editor)
int startX = (int)((widthSave - frame_vis.cols) / 2.0);
The warning(s) I am getting
Warning C26451 Arithmetic overflow: Using operator '-' on a 4 byte value and then casting the result to a 8 byte value.
Cast the value to the wider type before calling operator '-' to avoid overflow (io.2)
I have looked for it in all available forums, which led me to try "Casting" in varies ways (one by one, and all possible combinations) the 3 operands in that line to make sure they are 8 byte values. I can not get out of this hole!.. sometimes I can bypass this line, particularly when I used
int startX = (int)(( intptr_t(widthSave) - intptr_t(frame_vis.cols)) / intptr_t(2.0));
then I get 'just' 35 Warnings, meaning the other .cpp libraries 'involved', keep comming back with the same
Warning C26451: Arithmetic overflow...
Thanks for your feedback
fabianc20Fri, 01 Nov 2019 22:19:44 -0500http://answers.opencv.org/question/220859/Accuracy of cv2.aruco.estimatePoseSingleMarkershttp://answers.opencv.org/question/220644/accuracy-of-cv2arucoestimateposesinglemarkers/Hello everyone,
I have been trying to estimate the accuracy of cv2.aruco.estimatePoseSingleMarker in Python. To do so, I have generated a pixel picture with a perfectly centered AprilTag. I also created a synthetic camera matrix assuming no astigmatism and a perfectly centered focal point (cx,cy). Distortion Coefficients were set to zero. Using this synthetic data, if have been running the pose estimation.
Using this synthetic data, we should have an optical axis that is perfectly aligned with the marker/image center.
I did expect a tvec with X/Y-coordinates close to zero. However, I keep getting X/Y-coordinates with values around 10 units of length (in my case: mm).
I also tried it with ArUco Markers: Same result.
Please find my code and output attached below.
Is my assumption, that X/Y should be close to zero, correct? If so, what might be the reason for this offset? Is it related to the PnP Algorithm or is there a bug in my code?
Thanks in advance !
import cv2 as cv
import numpy as np #imports
dictionary = cv.aruco.getPredefinedDictionary(cv.aruco.DICT_APRILTAG_16H5)
para = cv.aruco.DetectorParameters_create()
para.cornerRefinementMethod = cv.aruco.CORNER_REFINE_APRILTAG
para.aprilTagDeglitch = 0
para.aprilTagMinWhiteBlackDiff = 30
para.aprilTagMaxLineFitMse = 20
para.maxErroneousBitsInBorderRate = 0.35
para.errorCorrectionRate = 1.0
para.minMarkerPerimeterRate = 0.05
para.maxMarkerPerimeterRate = 4
para.polygonalApproxAccuracyRate = 0.05
para.minCornerDistanceRate = 0.05
para.aprilTagCriticalRad = 0.1745329201221466 *6
para.aprilTagMinClusterPixels = 5 #set dictionary and parameter object
img = cv.imread("testimg.png")
camMat = np.array([[14492.753623188406, 0, 1024],[0, 14492.753623188406, 1224],[0, 0, 1]])#camera matrix for f = 50mm with chipsize = 0.00345 mm and 2048x2448px
distCoeffs = np.ndarray([0]) #distortion coefficients
marker_length = 30.00 #mm
corners,ids = cv.aruco.detectMarkers(img,dictionary,parameters = para)[:2]
corners = np.array(corners)
pose = cv.aruco.estimatePoseSingleMarkers(corners,marker_length,camMat,distCoeffs)
tvec = pose[1][0][0].tolist()
print(tvec) #get tvec
My output is:
Tvec for Apriltag Detection = [10.714285714225264, -10.714285714225268, 776.397515563502]
Tvec for ArUco Detection = [10.689704341562148, -10.743286819239351, 776.5576507340221]
Used Picture:
[C:\fakepath\testimg.png](/upfiles/15723624961531042.png)
My system is running:
Opencv: 4.1.1,
Python 3.7.3,
Windows 10cv_usr_mjiTue, 29 Oct 2019 10:06:41 -0500http://answers.opencv.org/question/220644/solvePnP object pose for Omnidirectional modelhttp://answers.opencv.org/question/197309/solvepnp-object-pose-for-omnidirectional-model/ I'm interested in pose estimation using slightly different sensors called omnidirectional camera, they are based on `Unified Omnidirectional Model` to project 3D points into image plane.
`Opencv` has a function called `solvePnP` to find the pose of an object using couple of 2D-3D points for standard cameras.
My question is if there is a similar function for those particular cameras? in `opencv` or any other library.ROSpioneerMon, 13 Aug 2018 04:10:36 -0500http://answers.opencv.org/question/197309/ArUco orientation using the function aruco.estimatePoseSingleMarkers()http://answers.opencv.org/question/215377/aruco-orientation-using-the-function-arucoestimateposesinglemarkers/Hi everyone!
I'm trying to program a python app that determine the position and orientation of an aruco marker. I calibrated the camera and everything and I used *aruco.estimatePoseSingleMarkers* that returns the translation and rotation vectors.
The translation vector works fine but I don't understand how the rotation vector works. I took some picture to illustrate my problem with the "roll rotation":
Here the rotation vector is approximately [in degree]: [180 0 0]
![image description](/upfiles/15626850347225475.png)
Here the rotation vector is approximately [in degree]: [123 -126 0]
![image description](/upfiles/15626851829885092.png)
And here the rotation vector is approximately [in degree]: [0 -180 0]
![image description](/upfiles/15626853815019584.png)
And I don't see the logic in these angles. I've tried the other two rotations (pitch and yaw) and there appear also "random". So if you have an explication I would be very happy :) lamaaTue, 09 Jul 2019 10:28:00 -0500http://answers.opencv.org/question/215377/Could the Real Time pose estimation tutorial be used for a custom shape object ?http://answers.opencv.org/question/210125/could-the-real-time-pose-estimation-tutorial-be-used-for-a-custom-shape-object/ The tutorial is here: https://docs.opencv.org/3.0-beta/doc/tutorials/calib3d/real_time_pose/real_time_pose.html
and it says: "The application starts up loading the 3D textured model in YAML file format". So I could use any object shape I could given I have the data for it?
ThankstrttrtMon, 11 Mar 2019 13:05:15 -0500http://answers.opencv.org/question/210125/SolvePnp doesn't give the same result with four points and more pointshttp://answers.opencv.org/question/209709/solvepnp-doesnt-give-the-same-result-with-four-points-and-more-points/ I'm computing the pose of the camera using `SolvePnP` function of `OpenCv` library and I'm facing the issue that when I give four point it gives me slightly better result than if I give more points. Here is an example of the code with some points as an example:
#include <opencv2/opencv.hpp>
int main(int argc, char **argv)
{
//with 11 points
std::vector<cv::Point2f> image_points_eleven = {
cv::Point2f(905.63, 694.37),
cv::Point2f(936.403, 696.781),
cv::Point2f(988.424, 700.553),
cv::Point2f(1020.02, 702.846),
cv::Point2f(1016.45, 741.839),
cv::Point2f(1012.79, 781.955),
cv::Point2f(1009.06, 822.609),
cv::Point2f(951.48, 815.937),
cv::Point2f(894.528, 810.86),
cv::Point2f(898.26, 772.418),
cv::Point2f(901.867, 732.744) };
std::vector<cv::Point3f> model_points_eleven = {
cv::Point3f(-155.32,155.32, 0),
cv::Point3f(-70.6,155.32, 0),
cv::Point3f(70.6,155.32, 0),
cv::Point3f(155.32,155.32, 0),
cv::Point3f(155.32,52.95, 0),
cv::Point3f(155.32,-52.95, 0),
cv::Point3f(155.32,-158.85, 0),
cv::Point3f(0, -155.32, 0),
cv::Point3f(-155.32,-155.32, 0),
cv::Point3f(-155.32,-52.95, 0),
cv::Point3f(-155.32, 52.95, 0)};
//with 4 points
std::vector<cv::Point2f> image_points_four = {
cv::Point2f(921.808,714.396),
cv::Point2f(999.474,720.263),
cv::Point2f(992.486,800.519),
cv::Point2f(914.465,793.569) };
std::vector<cv::Point3f> model_points_four = {
cv::Point3f(-211.5 / 2.0f, 211.5 / 2.0f, 0),
cv::Point3f(-211.5 / 2.0f, 211.5 / 2.0f, 0),
cv::Point3f(-211.5 / 2.0f, 211.5 / 2.0f, 0),
cv::Point3f(-211.5 / 2.0f, 211.5 / 2.0f, 0)};
// Camera internals
cv::Mat camera_matrix = (cv::Mat_<double>(3,3) << 1296.2477, 0, 1028, 0 , 1296.2477, 771, 0, 0, 1);
cv::Mat dist_coeffs = (cv::Mat_<double>(1,5) << -0.12658285, 0.13909541, 0, 0, -0.040676277); // Assuming no lens distortion
// Output rotation and translation
cv::Mat rotation_vector; // Rotation in axis-angle form
cv::Mat translation_vector;
// Solve for pose 11 points
cv::solvePnP(model_points_eleven, image_points_eleven, camera_matrix, dist_coeffs, rotation_vector, translation_vector);
rotation_vector.convertTo(rotation_vector, CV_32F);
translation_vector.convertTo(translation_vector, CV_32F);
std::cout << "Rotation Vector 11pts: " << rotation_vector << std::endl;
std::cout << "Translation Vector 11pts: " << translation_vector << std::endl;
// Solve for pose 11 points
cv::solvePnP(model_points_four, image_points_four, camera_matrix, dist_coeffs, rotation_vector, translation_vector);
rotation_vector.convertTo(rotation_vector, CV_32F);
translation_vector.convertTo(translation_vector, CV_32F);
std::cout << "Rotation Vector 4pts: " << rotation_vector << std::endl;
std::cout << "Translation Vector 4pts: " << translation_vector << std::endl;
}
I do get:
Rotation Vector 11pts: [3.0777242; 0.13331571; -0.26817828]
Translation Vector 11pts: [-187.29686; -36.374325; 3410.5]
Rotation Vector 4pts: [1.5553488; 0.13531595; -0.19250046]
Translation Vector 4pts: [83.4842; -4.0712709; -163.0168]
Comparing to an external results (lazer), for me I should get more precision the more points I have or at least the same result, however it doesn't seem to be the case. I get better result with four points when comparing `X, Y and Z`
Is there something wrong with my code?
PS: `model_points_four` and `model_points_eleven` have the same origin point in 3D world and in the image.ROSpioneerSat, 02 Mar 2019 14:07:11 -0600http://answers.opencv.org/question/209709/Calculating pose of aruco markers wrt a world framehttp://answers.opencv.org/question/193367/calculating-pose-of-aruco-markers-wrt-a-world-frame/I am trying to detect a drone using aruco markers. I have the following setup:
I have a downward facing camera which is fixed. I have an aruco marker on the ground, which defines the world frame. I have the rotation and translation vectors of the world frame wrt the camera frame (using estimatePoseSingleMarkers). I store these vectors for use later.
I have a drone with an aruco marker on top of it. The marker is offset by around 3 cm from the center of the drone (let's call this distance markerOffset). In my main code, I use estimatePoseSingleMarkers to calculate the rotation and translation vectors of the drone marker in the camera frame.
I have the following questions:
1. How do I compute the translation vector of the drone marker in the world frame? I have computed the rotation matrix by using Rodrigues() and multiplying the two rotation matrices. However, I am not getting accurate results with the translation vector
2. How do I correct for the markerOffset distance to find the translation vector of the drone in the world frame?
The following picture can help set the story straight.
![image description](/upfiles/15284593746652606.jpg)kmathFri, 08 Jun 2018 07:03:21 -0500http://answers.opencv.org/question/193367/Aruco Markers point position estimationhttp://answers.opencv.org/question/189278/aruco-markers-point-position-estimation/I need to detect and track an area of fixed size (let's say 1mt by 1mt).
I am using 4 sets of 3 markers (each one for a different corner) so that by reading the marker Id I can know which corner it belongs to. Each set of marker has an L disposition (oriented to match the corner shape).
If I can detect all the markers I am able to successfully identify the target area. But, if one or more markers are not detected (e.g. they are outside the frame) I would like to use the others to estimate the missing points.
Unfortuately pose estimation is good locally around the marker but gives very poor results if i try to estimate the position of farther points.
The code I am using at the moment is something like this
Marker m; //filled, imagine this is the bottom left marker and I want the top right one
CameraParameters camParams; //filled
Mat objPoint(1,3,CV_32FC1);
objPoint.at<float>(0,0) = 1.0f;
objPoint.at<float>(0,1) = 1.0f;
objPoint.at<float>(0,2) = 0.0f;
// From the center of the marker I want to move 1mt along x axis and 1mt along y axis
vector<Point2f> imagePoints;
projectPoints(objectPoints, m.Rvec, m.Tvec, camParams.CameraMatrix, camParams.Distorsion, imagePoints);
Point2f estimate = imagePoints[0];
At this point I expect estimate to be the point I wanted expressed in screen pixels. But the result is often very bad. I also tried using PoseTrakcers and "homemade tracking" by saving the previous positions of the markers and then by trying to predict the next.
Am I doing something wrong? Is there a better way?
I am using OpenCV 3.4.1 and Aruco 3.0.6.
Thanks in advance.
BobazThu, 12 Apr 2018 10:03:06 -0500http://answers.opencv.org/question/189278/cv::gpu::solvePnPRansachttp://answers.opencv.org/question/167076/cvgpusolvepnpransac/ I am using opencv 3.1.0 and I cannot find the gpu-based solvePnPRansac method. It is my understanding that the gpu::* methods and types were all moved somewhere else. Where can i find the equivalent of gpu::SolvePnpRansac?fab_cutMon, 17 Jul 2017 04:54:03 -0500http://answers.opencv.org/question/167076/How to use real_time_pose_estimationhttp://answers.opencv.org/question/160249/how-to-use-real_time_pose_estimation/ Hi everyone,
I am trying to use real_time_pose_estimation module in https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/calib3d
I generated cpp-tutorial-pnp_detection and cpp-tutorial-pnp_registration and use these in my own object.
However, the result video always cannot generate the correct solution.
My 3D textured model of the object to be detected is shown following:
![image description](/upfiles/14974724616086627.png)
![image description](/upfiles/14974724814749636.png)
However, my results always cannot detect the object as following:
![image description](/upfiles/14974725302574044.png)
Does there someone have experience that know the exact problem is?
I will be appreciated for your ideas.doubleWed, 14 Jun 2017 15:37:29 -0500http://answers.opencv.org/question/160249/Error in gauss-newton implementation for pose optimizationhttp://answers.opencv.org/question/147038/error-in-gauss-newton-implementation-for-pose-optimization/ Hello. I’m using a modified version of a gauss-newton method to refine a pose estimate using OpenCV. The unmodified code can be found here: [http://people.rennes.inria.fr/Eric.Marchand/pose-estimation/tutorial-pose-gauss-newton-opencv.html][1]
The details of this approach are outlined in the corresponding paper:
> Marchand, Eric, Hideaki Uchiyama, and Fabien Spindler. "Pose
> estimation for augmented reality: a hands-on survey." IEEE
> transactions on visualization and computer graphics 22.12 (2016):
> 2633-2651.
A PDF can be found here: [https://hal.inria.fr/hal-01246370/document][2]
The part that is relevant (Pages 4 and 5) are screencapped below:
[![Gauss-Newton Minimization][3]][3]
Here is what I have done. First, I’ve (hopefully) “corrected” some errors: (a) `dt` and `dR` can be passed by reference to `exponential_map()` (even though `cv::Mat` is essentially a pointer). (b) The last entry of each 2x6 Jacobian matrix, `J.at<double>(i*2+1,5)`, was `-x[i].y` but should be `-x[i].x`. (c) I’ve also tried using a different formula for the projection. Specifically, one that includes the focal length and principal point:
xq.at<double>(i*2,0) = cx + fx * cX.at<double>(0,0) / cX.at<double>(2,0);
xq.at<double>(i*2+1,0) = cy + fy * cX.at<double>(1,0) / cX.at<double>(2,0);
Here is the relevant code I am using, in its entirety (control starts at optimizePose3()):
void exponential_map(const cv::Mat &v, cv::Mat &dt, cv::Mat &dR)
{
double vx = v.at<double>(0,0);
double vy = v.at<double>(1,0);
double vz = v.at<double>(2,0);
double vtux = v.at<double>(3,0);
double vtuy = v.at<double>(4,0);
double vtuz = v.at<double>(5,0);
cv::Mat tu = (cv::Mat_<double>(3,1) << vtux, vtuy, vtuz); // theta u
cv::Rodrigues(tu, dR);
double theta = sqrt(tu.dot(tu));
double sinc = (fabs(theta) < 1.0e-8) ? 1.0 : sin(theta) / theta;
double mcosc = (fabs(theta) < 2.5e-4) ? 0.5 : (1.-cos(theta)) / theta / theta;
double msinc = (fabs(theta) < 2.5e-4) ? (1./6.) : (1.-sin(theta)/theta) / theta / theta;
dt.at<double>(0,0) = vx*(sinc + vtux*vtux*msinc)
+ vy*(vtux*vtuy*msinc - vtuz*mcosc)
+ vz*(vtux*vtuz*msinc + vtuy*mcosc);
dt.at<double>(1,0) = vx*(vtux*vtuy*msinc + vtuz*mcosc)
+ vy*(sinc + vtuy*vtuy*msinc)
+ vz*(vtuy*vtuz*msinc - vtux*mcosc);
dt.at<double>(2,0) = vx*(vtux*vtuz*msinc - vtuy*mcosc)
+ vy*(vtuy*vtuz*msinc + vtux*mcosc)
+ vz*(sinc + vtuz*vtuz*msinc);
}
void optimizePose3(const PoseEstimation &pose,
std::vector<FeatureMatch> &feature_matches,
PoseEstimation &optimized_pose) {
//Set camera parameters
double fx = camera_matrix.at<double>(0, 0); //Focal length
double fy = camera_matrix.at<double>(1, 1);
double cx = camera_matrix.at<double>(0, 2); //Principal point
double cy = camera_matrix.at<double>(1, 2);
auto inlier_matches = getInliers(pose, feature_matches);
std::vector<cv::Point3d> wX;
std::vector<cv::Point2d> x;
const unsigned int npoints = inlier_matches.size();
cv::Mat J(2*npoints, 6, CV_64F);
double lambda = 0.25;
cv::Mat xq(npoints*2, 1, CV_64F);
cv::Mat xn(npoints*2, 1, CV_64F);
double residual=0, residual_prev;
cv::Mat Jp;
for(auto i = 0u; i < npoints; i++) {
//Model points
const cv::Point2d &M = inlier_matches[i].model_point();
wX.emplace_back(M.x, M.y, 0.0);
//Imaged points
const cv::Point2d &I = inlier_matches[i].image_point();
xn.at<double>(i*2,0) = I.x; // x
xn.at<double>(i*2+1,0) = I.y; // y
x.push_back(I);
}
//Initial estimation
cv::Mat cRw = pose.rotation_matrix;
cv::Mat ctw = pose.translation_vector;
int nIters = 0;
// Iterative Gauss-Newton minimization loop
do {
for (auto i = 0u; i < npoints; i++) {
cv::Mat cX = cRw * cv::Mat(wX[i]) + ctw; // Update cX, cY, cZ
// Update x(q)
//xq.at<double>(i*2,0) = cX.at<double>(0,0) / cX.at<double>(2,0); // x(q) = cX/cZ
//xq.at<double>(i*2+1,0) = cX.at<double>(1,0) / cX.at<double>(2,0); // y(q) = cY/cZ
xq.at<double>(i*2,0) = cx + fx * cX.at<double>(0,0) / cX.at<double>(2,0);
xq.at<double>(i*2+1,0) = cy + fy * cX.at<double>(1,0) / cX.at<double>(2,0);
// Update J using equation (11)
J.at<double>(i*2,0) = -1 / cX.at<double>(2,0); // -1/cZ
J.at<double>(i*2,1) = 0;
J.at<double>(i*2,2) = x[i].x / cX.at<double>(2,0); // x/cZ
J.at<double>(i*2,3) = x[i].x * x[i].y; // xy
J.at<double>(i*2,4) = -(1 + x[i].x * x[i].x); // -(1+x^2)
J.at<double>(i*2,5) = x[i].y; // y
J.at<double>(i*2+1,0) = 0;
J.at<double>(i*2+1,1) = -1 / cX.at<double>(2,0); // -1/cZ
J.at<double>(i*2+1,2) = x[i].y / cX.at<double>(2,0); // y/cZ
J.at<double>(i*2+1,3) = 1 + x[i].y * x[i].y; // 1+y^2
J.at<double>(i*2+1,4) = -x[i].x * x[i].y; // -xy
J.at<double>(i*2+1,5) = -x[i].x; // -x
}
cv::Mat e_q = xq - xn; // Equation (7)
cv::Mat Jp = J.inv(cv::DECOMP_SVD); // Compute pseudo inverse of the Jacobian
cv::Mat dq = -lambda * Jp * e_q; // Equation (10)
cv::Mat dctw(3, 1, CV_64F), dcRw(3, 3, CV_64F);
exponential_map(dq, dctw, dcRw);
cRw = dcRw.t() * cRw; // Update the pose
ctw = dcRw.t() * (ctw - dctw);
residual_prev = residual; // Memorize previous residual
residual = e_q.dot(e_q); // Compute the actual residual
std::cout << "residual_prev: " << residual_prev << std::endl;
std::cout << "residual: " << residual << std::endl << std::endl;
nIters++;
} while (fabs(residual - residual_prev) > 0);
//} while (nIters < 30);
optimized_pose.rotation_matrix = cRw;
optimized_pose.translation_vector = ctw;
cv::Rodrigues(optimized_pose.rotation_matrix, optimized_pose.rotation_vector);
}
Even when I use the functions as given, it does not produce the correct results. My initial pose estimate is very close to optimal, but when I try run the program, the method takes a very long time to converge - and when it does, the results are very wrong. I’m not sure what could be wrong and I’m out of ideas. I’m confident my inliers are actually inliers (they were chosen using an M-estimator). I’ve compared the results from exponential map with those from other implementations, and they seem to agree.
So, where is the error in this gauss-newton implementation for pose optimization? I’ve tried to make things as easy as possible for anyone willing to lend a hand. Let me know if there is anymore information I can provide. Any help would be greatly appreciated. Thanks.
[1]: http://people.rennes.inria.fr/Eric.Marchand/pose-estimation/tutorial-pose-gauss-newton-opencv.html
[2]: https://hal.inria.fr/hal-01246370/document
[3]: https://i.stack.imgur.com/xPGia.pngsocraticMon, 08 May 2017 15:44:38 -0500http://answers.opencv.org/question/147038/Calculate odometry from camera poseshttp://answers.opencv.org/question/135672/calculate-odometry-from-camera-poses/I am doing pose estimation using ArUco markermaps. I followed the following procedure:
1. Created a markermap by printing and pasting markers in a room and recording their video and using it to create marker map (map.yml, map.log , and the point cloud file)
2. Use the generated markermap to run aruco tracking program. The tracking program outputs camera poses in real time in the form of three tvec and three rvec values.
Now, I want to keep record about how much the camera has moved in x and y positions (2d only) using the cam poses, so that I can generate my odometry on its basis. I am not sure how to interpret or use the tvec and rvec values for this purpose. Any ideas on this ? Thanks.
Note: I am a beginner in pose estimation, so If I seem to be going wrong in my approach, please mention it. b2meerFri, 24 Mar 2017 08:28:27 -0500http://answers.opencv.org/question/135672/In real_time_pose_estimation( samples given in opencv 3.1.0) no errors but fails to detect the objecthttp://answers.opencv.org/question/119146/in-real_time_pose_estimation-samples-given-in-opencv-310-no-errors-but-fails-to-detect-the-object/
1. Registered the 3D textured model of the object to be detected in a .yml file .[keyPoints.png](/upfiles/14818095612197492.png)
2. Storing the 3D information .[registration.png](/upfiles/14818097274279699.png)
3. In detection video plays but fails to detect the object.[detection.png](/upfiles/14818095352504625.png)
As am not good in understanding algorithms I need to know where am going wrong.joseph3010Thu, 15 Dec 2016 07:51:52 -0600http://answers.opencv.org/question/119146/As I am unable to understand formula/theory/details of this function details::focalsFromHomography () ? Can someone help me understanding or getting focals form Homography ?http://answers.opencv.org/question/102249/as-i-am-unable-to-understand-formulatheorydetails-of-this-function-detailsfocalsfromhomography-can-someone-help-me-understanding-or-getting-focals/ Hello,
I am unable to understand formula used in this function **details::focalsFromHomography ()** [here].
(http://docs.opencv.org/2.4/modules/stitching/doc/autocalib.html)
Can someone help me in getting proper theory behind formula used?srbhTue, 13 Sep 2016 10:41:23 -0500http://answers.opencv.org/question/102249/Entering >5 points into "findEssentialMat"?http://answers.opencv.org/question/89595/entering-5-points-into-findessentialmat/Hi,
I'm trying to figure out the relative pose (R, t matrices) of two cameras using two synchronized videos of the same scene viewed from different positions. As input I've selected points of the *same* object in corresponding frames of both videos, and findEssentialMat needs at least 5.
My question is: what happens if I feed in more than 5 problems? Ideally I'd like to feed in many more points than 5 points and get a more accurate estimate than just 5 points would give. That's what I hope. Otherwise I'd consider finding R,t from many pairs of 5 points of points, then taking the median or mean as the best estimate.
Thanks.gandalfsaxeMon, 07 Mar 2016 08:36:42 -0600http://answers.opencv.org/question/89595/Pose Estimation: Where is my error in reasoning?http://answers.opencv.org/question/79231/pose-estimation-where-is-my-error-in-reasoning/ Hi,
I am trying to do pose estimation. But I am clearly doing something wrong.
Let's say I have a pattern consisting of 4 Markers (A,B,C,D).
Each of these markers has an image coordinate and a pattern coordinate.
The origin of the pattern is the center of the polygon.
![Pattern upright](/upfiles/14498499273264979.png)
The image coordinates (x/y) are the following. (In a 1280x960 image)
> - Origin(616/814)
> - A(561/664)
>
> - B(702/838)
>
> - C(637/982)
>
> - D(520/755)
>
> Pattern coordinates (x/y/z)
>
> - Origin(0/0/0)
>
> - A(-12/32/0)
>
> - B(18/-5/0)
>
> - C(12/-36/0)
>
> - D(21/13/0)
Now it rotates by 90 degrees, but my coordinate system does not rotate with the pattern.
I am wondering what is wrong? Is it because the Z coordinate is always 0?
![Pattern horizontal](/upfiles/1449850109238967.png)
> - (x/y)
> - Origin(632/784)
> - A(718/812)
>
> - B(567/938)
>
> - C(441/909)
>
> - D(632/784)
>
> Pattern coordinates (x/y/z)
>
> - Origin(0/0/0)
>
> - A(32/12/0)
>
> - B(-4/18/0)
>
> - C(-35/11/0)
>
> - D(11/19/0)
I am using solvePnP like this
cv::solvePnP(patternPoints, imgPoints, cameraMatrix, distCoeffs, rvec, tvec);
Drawing the axis
//Method Call
pattern.drawAxis(image, camMatrix, distCoeffs, rvec, tvec,10);
//Implementation (taken from aruco.cpp)
void drawAxis(InputOutputArray _image, InputArray _cameraMatrix, InputArray _distCoeffs,
InputArray _rvec, InputArray _tvec, float length) {
CV_Assert(_image.getMat().total() != 0 &&
(_image.getMat().channels() == 1 || _image.getMat().channels() == 3));
CV_Assert(length > 0);
// project axis points
vector< Point3f > axisPoints;
axisPoints.push_back(Point3f(0, 0, 0));
axisPoints.push_back(Point3f(length, 0, 0));
axisPoints.push_back(Point3f(0, length, 0));
axisPoints.push_back(Point3f(0, 0, length));
vector< Point2f > imagePoints;
projectPoints(axisPoints, _rvec, _tvec, _cameraMatrix, _distCoeffs, imagePoints);
// draw axis lines
line(_image, imagePoints[0], imagePoints[1], Scalar(0, 0, 255), 3);
line(_image, imagePoints[0], imagePoints[2], Scalar(0, 255, 0), 3);
line(_image, imagePoints[0], imagePoints[3], Scalar(255, 0, 0), 3);
}
RVQMon, 14 Dec 2015 02:34:33 -0600http://answers.opencv.org/question/79231/relative pose between two imageshttp://answers.opencv.org/question/75562/relative-pose-between-two-images/Is it possible to get a relative pose between two images?
When I got E => (R, t) and K shouldn't it be possible to calculate the location of a point in the second image, found in the first image.
I know you need to know the scale factor for full reconstruction but do I really need it two if I just want a relative pose.
And lets say when there is only a rotation then a least the scaling factor should not be interesting?!
but why does it then be in this formula with R combined?
s2* m'2 = R * s1 * m'1 + tMJFri, 06 Nov 2015 10:27:51 -0600http://answers.opencv.org/question/75562/Units of Rotation and translation from Essential Matrixhttp://answers.opencv.org/question/66839/units-of-rotation-and-translation-from-essential-matrix/ I obtained my Rotation and translation matrices using the SVD of E. I used the following code to do so :
SVD svd(E);
Matx33d W(0, -1, 0, 1, 0, 0, 0, 0, 1);
Mat_<double> R = svd.u * Mat(W) * svd.vt;
Mat_<double> t = svd.u.col(2);
Matx34d P1(R(0, 0), R(0, 1), R(0, 2), t(0), R(1, 0), R(1, 1), R(1, 2), t(1), R(2, 0), R(2, 1), R(2, 2), t(2));
I want to know what are the units of R and t ? When I calibrated the cameras I got the baseline distance in **cm** , So is the translation vector units also in **cm** ? Also what are the units of R?
P.S : I have read the following question on [Stackoverflow](https://stackoverflow.com/questions/16856177/what-will-be-the-translation-vector-unit-from-essential-matrix/16870447#16870447) , but it had actually confused me as no answer was accepted. Also I wanted to know if the translation vector I got is it in the world frame?(Is it the real world distance moved from the initial position)
EDIT 1:
I have also observed that the values are normalized for the translation vector i.e x^2+y^2+z^2 is nearly 1. So how do I get the actual vector?
EDIT 2: I have read the following question in [Stackoverflow](https://stackoverflow.com/questions/3678317/t-and-r-estimation-from-essential-matrix?rq=1) and I think I will be implementing it. Don't know if there is any better way.Sai KondaveetiWed, 22 Jul 2015 22:11:34 -0500http://answers.opencv.org/question/66839/