OpenCV Q&A Forum - RSS feedhttp://answers.opencv.org/questions/OpenCV answersenCopyright <a href="http://www.opencv.org">OpenCV foundation</a>, 2012-2018.Wed, 13 Feb 2019 00:55:16 -0600Pose estimation from arbitrary imageshttp://answers.opencv.org/question/208802/pose-estimation-from-arbitrary-images/ I am using an arbitrary image and using the [template matching / homography][1] example provided by opencv to find the known image in the another image and find bounding boxes. I would further like to estimate the pose of the camera using the solvePnP or similar method. However I am confused as to how I can get the objectPoints in 3-D space and their respective 2D translations in camera image.
[![Detected image][2]][2] [![Original Image][3]][3]
How can I use [cv::solvePnP][4] to get the pose ?
I have the calibrated the camera and have the distortion coefficients, and the camera matrix. How can I generate the objects points (3D model) from the following original image and get the same points in 2D in the detected image?
[1]: https://docs.opencv.org/2.4/doc/tutorials/features2d/feature_homography/feature_homography.html
[2]: https://i.stack.imgur.com/wpPXO.png
[3]: https://i.stack.imgur.com/GfVXb.png
[4]: https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#solvepnpvmandkeWed, 13 Feb 2019 00:55:16 -0600http://answers.opencv.org/question/208802/Estimate object pose using multiple camerashttp://answers.opencv.org/question/208716/estimate-object-pose-using-multiple-cameras/Hello!
***TL;DR
I need a function similar to solvePnP(), but that would be able to estimate the pose of a model using information from multiple cameras instead of only one camera***
I am trying to find the pose (rotation and translation) of a simple object covered with markers, using n cameras placed around the object.
The pose of each camera is known: I already have a matrix Ci for each camera i such as for a point X=(x,y,z,1) in real world coordinates, Pi*X gives me the coordinates of that point in the camera's coordinate system.
The object I am trying to estimate is composed of m points, and I know the position of each of them in the object's coordinate system.
I am already able to find the object's coordinates in the plane of each cameras.
So if I put all this together, for each point j seen in a camera i I get this:
**sij * Pij = Ci * A * Xj**
where:
**sij** is an unknown scalar (it is here because we don't know how far from the camera the point found is) that multiplies the projection of the point j on the camera i *(unknown)*
**Pij** is the coordinates of the point j projected on the camera i: (x',y',1)T *(known)*
**Ci** is the matrix that describes the rotation and translation of the camera i *(known)*
**A** is the matrix I'm looking for, it describes the transformation between the object's coordinate system and real world coordinates *(unknown)*
**Xj** is the point j in the object's coordinate system: (x,y,z,1)T *(known)*
I will typically see 4 different points on 3 different cameras (the 12 points found are all differents), which would give me a set of 12 of those linear systems.
How do I find the matrix A that satisfies the best this set of linear systems ?
This problem looks like something that could be solved using DLT (https://en.wikipedia.org/wiki/Direct_linear_transformation), but I'm not able to transform my systems to fit the form shown on this wikipedia page.
*My question is similar to this one : http://answers.opencv.org/question/131660/multi-view-solvepnp-routine/,
but the answer there does not solve my problem because it requires that the points used to resolve the pose of the model are seen in multiple cameras.*Nick_Mon, 11 Feb 2019 15:46:00 -0600http://answers.opencv.org/question/208716/projectPoints functionality questionhttp://answers.opencv.org/question/96474/projectpoints-functionality-question/ I'm doing something similar to the tutorial here: http://docs.opencv.org/3.1.0/d7/d53/tutorial_py_pose.html#gsc.tab=0 regarding pose estimation. Essentially, I'm creating an axis in the model coordinate system and using ProjectPoints, along with my rvecs, tvecs, and cameraMatrix, to project the axis onto the image plane.
In my case, I'm working in the world coordinate space, and I have an rvec and tvec telling me the pose of an object. I'm creating an axis using world coordinate points (which assumes the object wasn't rotated or translated at all), and then using projectPoints() to draw the axes the object in the image plane.
I was wondering if it is possible to eliminate the projection, and get the world coordinates of those axes once they've been rotated and translated. To test, I've done the rotation and translation on the axis points manually, and then use projectPoints to project them onto the image plane (passing identity matrix and zero matrix for rotation, translation respectively), but the results seem way off. How can I eliminate the projection step to just get the world coordinates of the axes, once they've been rotation and translated? Thanks! bfc_opencvTue, 14 Jun 2016 21:19:07 -0500http://answers.opencv.org/question/96474/Solvepnp for fisheyehttp://answers.opencv.org/question/208634/solvepnp-for-fisheye/Hi,
I was following a [page](http://answers.opencv.org/question/67356/is-there-a-solvepnp-function-for-the-fisheye-camera-model/) to use solvepnp with a fisheye model. However, the answer suggests using the same camera matrix for the last input of undistortPoints function. This last input corresponds to the camera matrix in the new or rectified frame. How can we consider this matrix to be the same as the original camera matrix? Also, how can we use the rvec and tvec would correspond to undistorted corner points. Will this be the same as the rvec and tvec for the originally distorted image?
Thanks!svermaSun, 10 Feb 2019 19:22:19 -0600http://answers.opencv.org/question/208634/SolvePNP. Compensate for head rotationhttp://answers.opencv.org/question/208438/solvepnp-compensate-for-head-rotation/ Hi all!
I need to orientate detected face points to parallel with camera view.
I use the following code:
public void CalculateEulerAngles(int faceIndex)
{
var objectPoints = new List<Point3f>
{
...
};
var marks = Faces[faceIndex].Marks;
var imagePoints = new List<Point2f>
{
...
};
var focalLength = Image.Cols;
var center = new Point2f(Image.Cols / 2f, Image.Rows / 2f);
var cameraMatrix = new double[,] {{focalLength, 0, center.X}, {0, focalLength, center.Y}, {0, 0, 1}};
double[] rvec, tvec;
Cv2.SolvePnP(objectPoints, imagePoints, cameraMatrix, null, out rvec, out tvec);
double[,] rvecRodrigues;
Cv2.Rodrigues(rvec, out rvecRodrigues);
double[] eulerAngles;
double[,] camMatrix;
GetEulerAngles(rvecRodrigues, out eulerAngles, out camMatrix);
//
Point2f[] points;
double[,] jacobian;
Cv2.ProjectPoints(objectPoints, rvec, tvec, cameraMatrix, null, out points, out jacobian);
for (int i = 0; i < points.Length; i++)
Cv2.Circle(Image, (int)points[i].X, (int)points[i].Y, 5, Scalar.Aqua);
//
}
void GetEulerAngles(double[,] rvec, out double[] euler, out double[,] camMatrix)
{
double[] transVect, eulerAngles;
double[,] cameraMatrix,rotMatrix,rotMatrixX,rotMatrixY,rotMatrixZ;
double[,] projMatrix =
{
{rvec[0, 0], rvec[0, 1], rvec[0, 2], 0},
{rvec[1, 0], rvec[1, 1], rvec[1, 2], 0},
{rvec[2, 0], rvec[2, 1], rvec[2, 2], 0}
};
Cv2.DecomposeProjectionMatrix(projMatrix, out cameraMatrix, out rotMatrix, out transVect,
out rotMatrixX, out rotMatrixY, out rotMatrixZ, out eulerAngles);
euler = eulerAngles;
camMatrix = cameraMatrix;
}
But, I don't know how I can rotate points use calculated Euler angles to orientate parallel with camera view, so points always "flatted" to camera. Thanks guysmachinecoreTue, 05 Feb 2019 11:26:01 -0600http://answers.opencv.org/question/208438/SolvePnp origin pointhttp://answers.opencv.org/question/207232/solvepnp-origin-point/ Hello to everyone,
i am currently estimating the pose of a calibration chessboard using the functions findChessboardCorners and solvePnP for retrieving the spatial transform.
Currently it seems like the origin of the transform is the bottom-right corner of the chessboard, is there a way to make the center of the chessboard the origin point of the transformation?
I know i can do it afterwards knowing the geometric properties of the chessboard, i was just wondering if there is a better way directly in OpenCV.
ThanksmuttistefanoThu, 17 Jan 2019 05:06:13 -0600http://answers.opencv.org/question/207232/Unstable solvePnP in Iterative modehttp://answers.opencv.org/question/205246/unstable-solvepnp-in-iterative-mode/Hi,
we currently investigating really unstable results when applying solvePnP on our parameters. Under certain circumstances the complete translation vector is flipping into the wrong direction when slight changes to the focal point is applied.
Our minimal python example:
import numpy as np
import cv2 as cv
import platform
print('numpy:', np.__version__)
print('opencv:', cv.__version__)
print('architecture:', platform.architecture())
class Vector2d:
def __init__(self, X=0, Y=0):
self.X = X
self.Y = Y
def GetIntrinsicParametersMatrix(focalLength, principalPoint):
return np.array([
[focalLength.X, 0, principalPoint.X ],
[ 0, focalLength.Y, principalPoint.Y ],
[ 0, 0, 1 ]
], dtype=np.float32)
def GetDistortionCoefficients(calibrationData):
Radial = calibrationData['Radial']
Tangential = calibrationData['Tangential']
if (Radial[3] != 0 or Radial[4] != 0 or Radial[5] != 0):
return np.array([Radial[0], Radial[1], Tangential[0], Tangential[1], Radial[2], Radial[3], Radial[4], Radial[5]], dtype=np.float32)
elif Radial[2] != 0:
return np.array([Radial[0], Radial[1], Tangential[0], Tangential[1], Radial[2]], dtype=np.float32)
else:
return np.array([Radial[0], Radial[1], Tangential[0], Tangential[1]], dtype=np.float32)
if __name__ == '__main__':
imagePoints = [
[2590.57153, 1270.71423],
[2233.14282, 1032.85718],
[2945.71436, 1034.28577],
[1749.85718, 2218],
[3423.85718, 2219.85718]
]
modelPoints = [
[0, 15, 5.75],
[-23.5, 0, 0],
[23.5, 0, 0],
[-56.2, 78, 11.9],
[56.2, 78, 11.9]
]
calibrationData = {
'Radial': [ -0.0523676442172271, -0.335562126146013, 2.63059804821136, 0.0523676442003532,0.33556208204044, -2.63062713657836 ],
'Tangential' : [ 0, 0 ]
}
distortionCoefficients = GetDistortionCoefficients(calibrationData)
ppNormal = Vector2d(2591.661, 1728.591)
ppStrange = Vector2d(2591.66102211408, 1728.59081758821)
focalLength = Vector2d(8343.64179030465, 8344.73016777231)
# intrinsicMatrix = GetIntrinsicParametersMatrix(focalLength, ppNormal)
intrinsicMatrix = GetIntrinsicParametersMatrix(focalLength, ppStrange)
rvec = np.zeros([3, 1], dtype=np.double)
tvec = np.zeros([3, 1], dtype=np.double)
retval, rvec, tvec = cv.solvePnP(
objectPoints = np.array(modelPoints, dtype=np.double),
imagePoints = np.ascontiguousarray(np.array(imagePoints, dtype=np.double)[:,:2]).reshape((len(imagePoints),1,2)),
cameraMatrix = intrinsicMatrix,
distCoeffs = distortionCoefficients,
rvec = None,
tvec = None,
useExtrinsicGuess = False,
flags = cv.SOLVEPNP_ITERATIVE
)
# Expected around 550 but actually -551.69314282
print('tvec', tvec[2])
When using `ppStrange` the distance is totally flipped over, when using the rounded value `ppNormal` it's the expected (measured) somewhat around 550 mm. Switching to `SOLVEPNP_EPNP` is stabilizing the behavior, but seems to be less precise.
Any idea what causes the error and how to prevent or workaround this?
Tested with opencv 3.1.0.5 or 3.4.4. BTW.: Same behavior in c# with emgucv.
Edit:
Overview:
With:
`ppNormal = Vector2d(2591.661, 1728.591)`: 550.4662363
`ppStrange = Vector2d(2591.66102211408, 1728.59081758821)`: -551.69314282MaxDatenWed, 12 Dec 2018 10:14:55 -0600http://answers.opencv.org/question/205246/SolvePnP vs Calibrate camerahttp://answers.opencv.org/question/205240/solvepnp-vs-calibrate-camera/I have a very quick question, in case I'll try to generate a simple example.
I'm writing some alignment code, given instrinsics and distortion parameters.
The idea is to pre-estimate intrinsics and later, given a couple of images from different poses estimate the extrinsics.
For this job I'm running the opencv examples on the subject. For the camera calibration the meat of the code is `calibrate_camera`, such function according to the doc will estimate everything including extrinsics per pose.
It is raccomended however to use `solvePnP` to retrieve the extrinsics if the intrinsics are already given.
It is also mentioned in the doc that `calibrate_camera` also calls `solvePnP` at some point.
However when I run `solvePnP` for an image pair that I have also used to find intrinsics I get different results for the extrinsics.
To summarize, to find the intrinsics and distortion of the camera I'm using the code provided by [this example](https://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/camera_calibration.html), given these intrinsics and distortion I'm trying to estimate the extrinsics using the a `solvePnP` call (I need to call it two times, one per pose and with some linear algebra I can get the relative pose).
However as I said I'm getting different results, and I can't figure why (quite different results).
What I'm getting though is valid, since I can construct a valid essential matrix (I've tested this using the single value decomposition).user8469759Wed, 12 Dec 2018 09:05:36 -0600http://answers.opencv.org/question/205240/Calculate robot coordinates from measured chessbaord corners ( hand-eye calibration)http://answers.opencv.org/question/204910/calculate-robot-coordinates-from-measured-chessbaord-corners-hand-eye-calibration/Hello guys,
for my project I am trying to calibrate my depth camera with my robot-arm (hand-eye calibration). Therefore, I measured 8 corners of the cheassboard and got 8 pixel vectors and their corresponding robot 3D coordinates (using the gripper to point exactly at the corner). Then, I tried to calculate the transformation matrix. But the result is way off. Where's the mistake? Thank you very much!
Edit: Added the camera matrix
----------
img_pts = np.array(((237., 325.),
(242., 245.),
(248., 164.),
(318., 330.),
(322., 250.),
(328., 170.),
(398., 336.),
(403., 255.)), dtype=np.float32)
objectPoints = np.array(((-437., -491., -259.),
(-402., -457., -259.),
(-360., -421., -259.),
(-393., -534., -259.),
(-362., -504., -259.),
(-332., -476., -259.),
(-298., -511., -259.),
(-334., -546., -259.)), dtype=np.float32)
camera_matrix = np.array(((542.5860286838929, 0., 310.4749867256299),
(0., 544.2396851407029, 246.7577402755397),
(0., 0., 1.)), dtype=np.float32)
distortion = np.array([0.03056762860315778,
-0.1824835906443329,
-0.0007936359893356936,
-0.001735795279032343,
0])
ret, rvec, tvec = cv2.solvePnP(objectPoints, img_pts, camera_matrix, distortion)
rmtx, _ = cv2.Rodrigues(rvec)
T_gripper_cam = np.array([[rmtx[0][0], rmtx[0][1], rmtx[0][2], tvec[0]],
[rmtx[1][0], rmtx[1][1], rmtx[1][2], tvec[1]],
[rmtx[2][0], rmtx[2][1], rmtx[2][2], tvec[2]],
[0.0, 0.0, 0.0, 1.0]])
T_cam_gripper = np.linalg.inv(T_gripper_cam)
print(T_cam_gripper)
p = np.array((237., 325., 0.), dtype=np.float32)
p_new = np.array([p[0], p[1], p[2], 1])
objectPoint = np.dot(p_new, T_gripper_cam)[:3]
print(objectPoint)
# should be (-437., -491., -259.)
# --> actual = [ 33.57 -395.62 -64.46]
------------------------------------- Edit ----------------------------------------------------------
I think I might be on to something. According to this:
[Stackoverflow](https://stackoverflow.com/questions/12299870/computing-x-y-coordinate-3d-from-image-point)
I have to calculate the scaling factor. So i tried to do this in python:
uv = np.array(([237.], [325.], [1.]), dtype=np.float32)
rotInv = np.linalg.inv(rmtx)
camMatInv = np.linalg.inv(camera_matrix)
leftSideMat = np.dot(rotInv, np.dot(camMatInv, uv))
rightSideMat = np.dot(rotInv, tvec)
s = (295 + leftSideMat[2:] / rightSideMat[2:])
temp = (s * np.dot(camMatInv, uv))
tempSub = np.array(temp - tvec)
print(np.dot(rotInv, tempSub))
# should be (-437., -491., -259.)
# --> actual = [ 437.2 -501.3 -266.2]
Looks like I am pretty close. But its still way off in the Y direction. Is this the correct approach? Thank you!Mysterion46Fri, 07 Dec 2018 06:35:46 -0600http://answers.opencv.org/question/204910/creating a rotation matrix in OpenCVhttp://answers.opencv.org/question/204092/creating-a-rotation-matrix-in-opencv/ I am trying to recreate a rotation matrix in OpenCV using one of the direction vectors along the z-axes but have not been having any success with it. I am also not sure if this can be done in a unique fashion but I was hoping to get some help from the forum.
So, initially, I try and create the rotation matrix that I want to recover. I do this as following:
import cv2
import numpy
def generate_fake_opencv_calibrations(num_calibs=10):
calibs = list()
cm = np.asarray([[300.0, 0.0, 400], [0, 300.0, 700.0], [0., 0., 1.]])
for i in range(num_calibs):
image_c = np.random.rand(6, 2) * 100.0
world_c = np.random.rand(6, 3)
_, r, t = cv2.solvePnP(world_c, image_c, cm, None, flags=cv2.SOLVEPNP_ITERATIVE)
# Create the rotation matrix
r, _ = cv2.Rodrigues(r)
calibs.append(r)
return calibs
So, the rotation matrix has all the properties that the rows and columns are orthonormal wrt to each other. So, what I was hoping to do is recreate this rotation matrix when I have the direction vector for the z-axes.
So, I have:
z = c[:, 2]
Now I wrote a function to create the other two axes as:
def create_orthonormal_basis(v):
v = v / np.linalg.norm(v)
if v[0] > 0.9:
b1 = np.asarray([0.0, 1.0, 0.0])
else:
b1 = np.asarray([1.0, 0.0, 0.0])
b1 -= v * np.dot(b1, v)
b1 *= np.reciprocal(np.linalg.norm(b1))
b2 = np.cross(v, b1)
return b1, b2, v
I can then create the matrix as:
x, y, z = create_orthonormal_basis(z)
mat = np.asarray([[x[0], y[0], z[0]],
[x[1], y[1], z[1]],
[x[2], y[2], z[2]]])
I was expecting this matrix to map a given point to approximately the same location, however this was not the case. So for a random case, I am getting the following:
For the input matrix, given by:
[-0.5917787 -0.69902414 0.40145141]
[ 0.76717701 -0.64127655 0.01427625]
[ 0.24746193 0.31643267 0.91576905]]
The output is:
[ 0.91588032 0. 0.40145141]
[-0.00625761 0.99987851 0.01427625]
[-0.40140263 -0.01558747 0.91576905]]
I take a random input like:
[0.33385406 0.91243684 0.33755828]
and map it using the original and reconstructed matrices and the output is quite different:
Original: [-0.69986985 -0.32418012 0.68046643]
Reconstructed: [0.44128362 0.91505592 0.16089295]
pamparanaMon, 26 Nov 2018 09:38:03 -0600http://answers.opencv.org/question/204092/Relative marker positionhttp://answers.opencv.org/question/201475/relative-marker-position/Hello, I am using solvePNP to compute the poses relative to two planar markers.
The markers 0-1 are identical and both visible in the same frame, say more or less like two checkerboards. My final goal is that I want to calculate the marker 1 coordinates relative to marker 0.
I have pose `P0=[R0 t0]`, pose `P1=[R1 t1]`. Please note that the poses are calculated for the same image, so they are in fact the same pose relative to two different markers.
I have calculated those based on equal 3D point coordinates for both poses, since the markers are identical.
Now I want to get the 3D transformation that brings 3D points of one marker to 3D points on the other marker, so that I know the relative positions of the markers.
I know I can calculate a transform which converts the pose relative to marker 1 to a pose relative to marker 0 as such:
T01 = [R0 t0] [R1-1 -R1-1*t1] = [R0*R1-1 -R0*R1-1*t1+t0]
With this I can transform a generic pose P2 relative to marker 1 and get the pose from marker 0:
P2_0=T01*P2.
How do I use this information to get the trasnform from marker 0 coordinates to marker 1?klaymanFri, 19 Oct 2018 05:47:53 -0500http://answers.opencv.org/question/201475/solvePnP and problems with perspectivehttp://answers.opencv.org/question/201261/solvepnp-and-problems-with-perspective/I feed 8 corners of the cube, but after reprojection sometimes (not for all images) the nearest point becomes the farthest and vice versa.
![image description](/upfiles/15397170586303682.jpg)
Can anybody explain the reason and how to cope with it?
An example of correct case:
![image description](/upfiles/15397753139440974.jpg)
The code:
float box_side_x = 6; //Centimetres
float box_side_y = 6;
float box_side_z = 6;
vector<Point3f> boxPoints;
//Fill the array of corners in object coordinates. x to right(view from camera), y down, z from camera.
vector<Vec3d> boxCorners(8);
Vec3d boxCorner;
float x, y, z;
for (int h = 0; h < 2; ++h) {
for (int j = 0; j < 2; ++j) {
for (int i = 0; i < 2; ++i) {
x = box_side_x * i;
y = box_side_y * j;
z = box_side_z * h;
boxPoints.push_back(Point3f(x, y, z)); //For solvePnP()
boxCorners[i + 2 * j + 4 * h] = {x, y, z}; //For calculating output
}
}
}
solvePnP(boxPoints, pointBuf, cameraMatrix, distCoeffs, rvec, tvec, false);
Mat rmat;
Rodrigues(rvec, rmat);
Mat Result;
float S = 1;
for (int h = 0; h < 2; ++h) {
for (int j = 0; j < 2; ++j) {
for (int i = 0; i < 2; ++i) {
boxCorner = boxCorners[i+2*j+4*h]; //In centimetres
Result = S * (rmat * Mat(boxCorner) + tvec);
ObjPoints[i + 2 * j + 4 * h] = (Vec3f)Result; //In centimetres
}
}
}
In different file ObjPoints -> BoxCorners
vector<float> X(8), Y(8), Z(8);
for (int i = 0; i < 8; i++) {
BoxCorner = BoxCorners[i]; //In centimetres
X[i] = K * (BoxCorner[0] + Lx);
Y[i] = K * (BoxCorner[1] + Ly);
Z[i] = K * (BoxCorner[2] + Lz);
}
Scalar color = CV_RGB(0, 0, 200), back_color = CV_RGB(0, 0, 100);
int thickness = 2;
namedWindow("Projection", WINDOW_AUTOSIZE);
Point pt1 = Point(0, 0), pt2 = Point(0, 0);
pt1 = Point(X[7], Y[7]), pt2 = Point(X[6], Y[6]);
line(drawing, pt1, pt2, back_color, thickness);
pt1 = Point(X[7], Y[7]); pt2 = Point(X[5], Y[5]);
line(drawing, pt1, pt2, back_color, thickness);
pt1 = Point(X[7], Y[7]); pt2 = Point(X[3], Y[3]);
line(drawing, pt1, pt2, back_color, thickness);
pt1 = Point(X[0], Y[0]), pt2 = Point(X[1], Y[1]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[0], Y[0]); pt2 = Point(X[2], Y[2]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[0], Y[0]), pt2 = Point(X[4], Y[4]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[1], Y[1]); pt2 = Point(X[3], Y[3]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[3], Y[3]); pt2 = Point(X[2], Y[2]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[2], Y[2]); pt2 = Point(X[6], Y[6]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[6], Y[6]); pt2 = Point(X[4], Y[4]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[4], Y[4]); pt2 = Point(X[5], Y[5]);
line(drawing, pt1, pt2, color, thickness);
pt1 = Point(X[5], Y[5]); pt2 = Point(X[1], Y[1]);
line(drawing, pt1, pt2, color, thickness);
imshow("Projection", drawing);ya_ocv_userTue, 16 Oct 2018 14:13:19 -0500http://answers.opencv.org/question/201261/homography vs SolvePNP for pose detection, how and why?http://answers.opencv.org/question/201340/homography-vs-solvepnp-for-pose-detection-how-and-why/I have mutiple planar markers where I can detect 100-200 points each in a reliable manner.
In each frame I see one or two markers at most; I need to calculate the camera pose.
I am not sure whether I should use solvePNP or findhomography for the pose detection.
1) OpenCV homography page states that solvePnp should be used for planar objects as well, and findhomography is just for demo in terms of pose detection. See https://docs.opencv.org/3.4.1/d9/dab/tutorial_homography.html
On the other hand, solvePNP is using POSIT, right? Which should be less accurate on planar features, please correct me here if the implementation is actually taking care of the planar case. So, should I ever consider findHomography or not for pose estimation?
2) In case findhomography still makes sense, how should I use it? Should I use two frames and feed 2d-2d coordinates to the findhomography, or use the "3d object coordinates without Z" as the source?
Thanks for any hint.
klaymanWed, 17 Oct 2018 11:54:43 -0500http://answers.opencv.org/question/201340/Multiple markers along trajectoryhttp://answers.opencv.org/question/200942/multiple-markers-along-trajectory/ I have multiple planar markers in the scene. At any time I see one or two, sometimes three markers. I want to track camera pose on a trajectory which pans and rolls.
Right now I can track the pose with solvepnp for one marker at a time. I need to somehow "stitch" together the tracked poses for the other markers. For example, by calculating the "relative pose" on those frames where multiple markers are visible at the same time.
Do you have suggestions? Is this already supported by openCV? How can I build a smooth stitching?
klaymanThu, 11 Oct 2018 04:48:14 -0500http://answers.opencv.org/question/200942/solvePnP with a priori known pitch and rollhttp://answers.opencv.org/question/199943/solvepnp-with-a-priori-known-pitch-and-roll/How to correctly call solvePnP (for estimate the pose of a large ArUco board), if the board orientation (pitch and roll, not yaw) is known (from an IMU)?okalachevSat, 22 Sep 2018 13:59:48 -0500http://answers.opencv.org/question/199943/Augmented reality with Aruco and SceneKithttp://answers.opencv.org/question/199746/augmented-reality-with-aruco-and-scenekit/ I'm trying to make augmented reality demo project with simple 3d object in the center of marker. I need to make it with `OpenCV` and `SceneKit`.
My steps are:
- I obtain the corners of marker using `Aruco`
- with `cv::solvePnP` get the `tvec` and `rvec`.
- convert the `tvec` and `rvec` from `OpenCv's Coordinate System` to the `SceneKit Coordinate System`.
- apply the converted rotation and translation to the camera node.
The Problem is:
- The object is not centered on the marker. Rotation of object looks good. But is not positioned where it should be.
SolvePnp code:
cv::Mat intrinMat(3,3,cv::DataType<double>::type);
//From ARKit (ARFrame camera.intrinsics) - iphone 6s plus
intrinMat.at<double>(0,0) = 1662.49;
intrinMat.at<double>(0,1) = 0.0;
intrinMat.at<double>(0,2) = 0.0;
intrinMat.at<double>(1,0) = 0.0;
intrinMat.at<double>(1,1) = 1662.49;
intrinMat.at<double>(1,2) = 0.0;
intrinMat.at<double>(2,0) = 960.0 / 2;
intrinMat.at<double>(2,1) = 540.0 / 2;
intrinMat.at<double>(2,2) = 0.0;
double marker_dim = 3;
cv::Ptr<cv::aruco::Dictionary> dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_250);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
void *baseaddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
CGFloat width = CVPixelBufferGetWidth(pixelBuffer);
CGFloat height = CVPixelBufferGetHeight(pixelBuffer);
cv::Mat mat(height, width, CV_8UC1, baseaddress, 0); //CV_8UC1
cv::rotate(mat, mat, cv::ROTATE_90_CLOCKWISE);
std::vector<int> ids;
std::vector<std::vector<cv::Point2f>> corners;
cv::aruco::detectMarkers(mat,dictionary,corners,ids);
if(ids.size() > 0) {
cv::Mat colorMat;
cv::cvtColor(mat, colorMat, CV_GRAY2RGB);
cv::aruco::drawDetectedMarkers(colorMat, corners, ids, cv::Scalar(0,255,24));
cv::Mat distCoeffs = cv::Mat::zeros(8, 1, cv::DataType<double>::type); //zero out distortion for now
//MARK: solvepnp
std::vector<cv::Point3f> object_points;
object_points = {cv::Point3f(-marker_dim , marker_dim , 0),
cv::Point3f(marker_dim , marker_dim , 0),
cv::Point3f(marker_dim , -marker_dim , 0),
cv::Point3f(-marker_dim , -marker_dim , 0)};
std::vector<cv::Point_<float>> image_points = std::vector<cv::Point2f>{corners[0][0], corners[0][1], corners[0][2], corners[0][3]};
std::cout << "object points: " << object_points << std::endl;
std::cout << "image points: " << image_points << std::endl;
cv::Mat rvec, tvec;
cv::solvePnP(object_points, image_points, intrinMat, distCoeffs, rvec, tvec);
cv::aruco::drawAxis(colorMat, intrinMat, distCoeffs, rvec, tvec, 3);
cv::Mat rotation, transform_matrix;
cv::Mat RotX(3, 3, cv::DataType<double>::type);
cv::setIdentity(RotX);
RotX.at<double>(4) = -1; //cos(180) = -1
RotX.at<double>(8) = -1;
cv::Mat R;
cv::Rodrigues(rvec, R);
std::cout << "rvecs: " << rvec << std::endl;
std::cout << "cv::Rodrigues(rvecs, R);: " << R << std::endl;
R = R.t(); // rotation of inverse
std::cout << "R = R.t() : " << R << std::endl;
cv::Mat rvecConverted;
Rodrigues(R, rvecConverted); //
std::cout << "rvec in world coords:\n" << rvecConverted << std::endl;
rvecConverted = RotX * rvecConverted;
std::cout << "rvec scenekit :\n" << rvecConverted << std::endl;
Rodrigues(rvecConverted, rotation);
std::cout << "-R: " << -R << std::endl;
std::cout << "tvec: " << tvec << std::endl;
cv::Mat tvecConverted = -R * tvec;
std::cout << "tvec in world coords:\n" << tvecConverted << std::endl;
tvecConverted = RotX * tvecConverted;
std::cout << "tvec scenekit :\n" << tvecConverted << std::endl;
SCNVector4 rotationVector = SCNVector4Make(rvecConverted.at<double>(0), rvecConverted.at<double>(1), rvecConverted.at<double>(2), norm(rvecConverted));
SCNVector3 translationVector = SCNVector3Make(tvecConverted.at<double>(0), tvecConverted.at<double>(1), tvecConverted.at<double>(2));
std::cout << "rotation :\n" << rotation << std::endl;
transform_matrix.create(4, 4, CV_64FC1);
transform_matrix( cv::Range(0,3), cv::Range(0,3) ) = rotation * 1;
transform_matrix.at<double>(0, 3) = tvecConverted.at<double>(0,0);
transform_matrix.at<double>(1, 3) = tvecConverted.at<double>(1,0);
transform_matrix.at<double>(2, 3) = tvecConverted.at<double>(2,0);
transform_matrix.at<double>(3, 3) = 1;
TransformModel *model = [TransformModel new];
model.rotationVector = rotationVector;
model.translationVector = translationVector;
return model;
}
swift code:
func initSceneKit() {
let scene = SCNScene()
cameraNode = SCNNode()
let camera = SCNCamera()
camera.zFar = 1000
camera.zNear = 0.1
cameraNode.camera = camera
scene.rootNode.addChildNode(cameraNode)
let scnView = sceneView!
scnView.scene = scene
scnView.autoenablesDefaultLighting = true
scnView.backgroundColor = UIColor.clear
let box = SCNBox(width: 10, height: 10 , length: 10, chamferRadius: 0)
boxNode = SCNNode(geometry: box)
boxNode.position = SCNVector3(0,0,0)
scene.rootNode.addChildNode(boxNode)
sceneView.pointOfView = cameraNode
}
func initCamera() {
let device = AVCaptureDevice.default(AVCaptureDevice.DeviceType.builtInWideAngleCamera, for: .video, position: AVCaptureDevice.Position.back)
let deviceInput = try! AVCaptureDeviceInput(device: device!)
self.session = AVCaptureSession()
self.session.sessionPreset = AVCaptureSession.Preset.iFrame960x540
self.session.addInput(deviceInput)
let sessionOutput: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput()
let outputQueue = DispatchQueue(label: "VideoDataOutputQueue", attributes: [])
sessionOutput.setSampleBufferDelegate(self, queue: outputQueue)
self.session.addOutput(sessionOutput)
self.previewLayer = AVCaptureVideoPreviewLayer(session: self.session)
self.previewLayer.backgroundColor = UIColor.black.cgColor
self.previewLayer.videoGravity = AVLayerVideoGravity.resizeAspect
self.previewView.layer.addSublayer(self.previewLayer)
self.session.startRunning()
view.setNeedsLayout()
}
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
//QR detection
guard let transQR = OpenCVWrapper.arucoTransformMatrix(from: pixelBuffer) else {
return
}
DispatchQueue.main.async(execute: {
self.setCameraMatrix(transQR)
})
}
func setCameraMatrix(_ transformModel: TransformModel) {
cameraNode.rotation = transformModel.rotationVector
cameraNode.position = transformModel.translationVector
// cameraNode.transform = transformModel.transform
}
[image of my result](https://i.stack.imgur.com/Calxx.jpg)
Repo on github with my project: [https://github.com/danilovdorin/ArucoAugmentedReality](https://github.com/danilovdorin/ArucoAugmentedReality)dorindanilovWed, 19 Sep 2018 15:28:20 -0500http://answers.opencv.org/question/199746/how to take points InputArray objectPoints, InputArray imagePoints could you please explain in detail with an examplehttp://answers.opencv.org/question/198294/how-to-take-points-inputarray-objectpoints-inputarray-imagepoints-could-you-please-explain-in-detail-with-an-example/ solvePnP¶
Finds an object pose from 3D-2D point correspondences.
C++: bool solvePnP(InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArray rvec, OutputArray tvec, bool useExtrinsicGuess=false, int flags=ITERATIVE )
srinath14144Wed, 29 Aug 2018 10:36:57 -0500http://answers.opencv.org/question/198294/Rotation vector interpretationhttp://answers.opencv.org/question/197981/rotation-vector-interpretation/I use opencv cv2.solvePnP() function to calculate rotation and translation vectors. Rotation is returned as rvec [vector with 3DOF]. I would like to ask for help with interpreting the rvec.
As far as I understand rvec = the rotation vector representation:
- the rotation vector is the axis of the rotation
- the length of rotation vector is the rotation angle θ in radians [around axis, so rotation vector]
Rvec returned by solvePnP:
rvec =
[[-1.5147142 ]
[ 0.11365167]
[ 0.10590861]]
Then:
angle_around_rvec = sqrt(-1.5147142^2 + 0.11365167^2 + 0.10590861^2) [rad] = 1.52266 [rad] = 1.52266*180/3.14 [deg] = 87.286 [deg]
**1. Does 3 rvec components correspond to world coordinates? Or what are these directions?**
**2. Can I interpret the vector components as separate rotation angles in radians around components directions?**
My rvec components interpretation:
angle_around_X = -1.5147142 [rad] = -1.5147*180/3.14 [deg] = -86.83 [deg]
angle_around_Y = 0.11365167 [rad] = 0.11365167*180/3.14 [deg] = 6.52 [deg]
angle_around_Z = 0.10590861 [rad] = 0.10590861*180/3.14 [deg] = 6.07 [deg]
My usecase:
I have coordinates of four image points. I know the coordinates of these points in the real world. I know camera intrinsic matrix. I use PnP3 to get rotation and translation vector. From rotation matrix, I would like to find out what are the angles around fixed global/world axes: X, Y, Z. I am NOT interested in Euler angles. I want to find out how an object is being rotated around the fixed world coordinates (not it's own coordinate system).
I would really appreciate your help. I feel lost in rotation.
Thank you in advance.dziadygeThu, 23 Aug 2018 13:55:50 -0500http://answers.opencv.org/question/197981/SolvePnPRansac, all axes look great except for one.http://answers.opencv.org/question/196582/solvepnpransac-all-axes-look-great-except-for-one/I am running solvePnPRansac on an image dataset, with 2d feature points and triangulated 3d landmark points. It runs great, and the results in rotation, and in the forward and side axes, look great. The Y axis though, is completely wrong.
I am testing the output against the ground truth from the data set, and it goes up where it should go down, and drifts off the ground truth very quickly. The other axes stay locked on.
this strikes me as strange, how can it be correct for the other axes, and wrong for one? Surely that is not possible?
What could i possibly be doing wrong to make this happen? opencv 4.0 pre, windows 10.
cv::Mat inliers;
cv::Mat rvec = cv::Mat::zeros(3, 1, CV_64FC1);
int iterationsCount = 500; // number of Ransac iterations.
float reprojectionError = 2.0; //2.0 // maximum allowed distance to consider it an inlier.
float confidence = 0.95; // RANSAC successful confidence.
bool useExtrinsicGuess = false;
int flags = cv::SOLVEPNP_ITERATIVE;
int num_inliers_;
//points3D_t0
cv::solvePnPRansac(points3D_t0, points_left_t1, intrinsic_matrix, distCoeffs, rvec, translation_stereo,
useExtrinsicGuess, iterationsCount, reprojectionError, confidence,
inliers, flags);antithingSun, 29 Jul 2018 12:25:41 -0500http://answers.opencv.org/question/196582/solvePnPRansac, getting inliers from the 2d and 3d points?http://answers.opencv.org/question/196562/solvepnpransac-getting-inliers-from-the-2d-and-3d-points/ I am running solvePnPRansac, and after it runs, i need to get the inlier points from the input matrices. The inlier Mat seems to be a list of ints, rather than the 0 or 1 values that i would expect. What is the best way to run this:
cv::Mat points3D_t0; // Mat of 3d points
std::vector<Point2f> points_left_t1; // vector of 2d features
cv::solvePnPRansac(points3D_t0, points_left_t1, intrinsic_matrix, distCoeffs, rvec, tvec,
useExtrinsicGuess, iterationsCount, reprojectionError, confidence,
inliers, flags);
and then fill a `std::vector<Point2f>` and a `std::vector<Point3f>` with the matched inlier points that were used in the PnP solve?
Thank you.antithingSat, 28 Jul 2018 16:19:39 -0500http://answers.opencv.org/question/196562/Camera position in world coordinate is not working but object pose in camera co ordinate system is working properlyhttp://answers.opencv.org/question/194724/camera-position-in-world-coordinate-is-not-working-but-object-pose-in-camera-co-ordinate-system-is-working-properly/I am working on the camera (iphone camera) pose estimation for head mount device (Hololens) using LEDs as a marker, using the solvepnp. I have calibrated the camera below is the camera intrinsic parameters
/* approx model*/
double focal_length = image.cols;
Point2d center = cv::Point2d(image.cols/2,image.rows/2);
iphone_camera_matrix = (cv::Mat_<double>(3,3) << focal_length, 0, center.x, 0 , focal_length, center.y, 0, 0, 1); iphone_dist_coeffs = cv::Mat::zeros(4,1,cv::DataType<double>::type);
/* caliberated(usng opencv) model */
iphone_camera_matrix = (cv::Mat_<double>(3,3) <<839.43920487140315, 0, 240, 0, 839.43920487140315, 424, 0, 0, 1);
iphone_dist_coeffs = (cv::Mat_<double>(5,1) <<4.6476561543838640e-02, -2.0580084834071521, 0, 0 ,2.0182662261396342e+01);
usng solvpnp am able to get the proper object pose in camera co-ordinate system below is the code
cv::solvePnP(world_points, image_points, iphone_camera_matrix, iphone_dist_coeffs, rotation_vector, translation_vector, true, SOLVEPNP_ITERATIVE);
the ouput is
rotation_vector :
[-65.41956646885059;
-52.49185328449133;
36.82917796058498]
translation_vector :
[94.1158604375937;
-164.2178023980637;
580.5666657301058]
using this rotation_vector and translation_vector am visualizing pose by projecting the trivector whose points are
points_to_project :
[0, 0, 0;
20, 0, 0;
0, 20, 0;
0, 0, 20]
projectPoints(points_to_project, rotation_vector, translation_vector, iphone_camera_matrix, iphone_dist_coeffs, projected_points);
the output of projectedPoints given as
projected_points :
[376.88803, 185.15131;
383.05768, 195.77643;
406.46454, 175.12997;
372.67371, 155.56181]
which seems correct as shown below
![object pose in camera co-ordinate system](/upfiles/15302624392884724.png)
I try to find the camera pose in world/object coordinate system by using the transformation of rotation_vector and translation_vector given by solvepnp as
cv::Rodrigues(rotation_vector, rotation_matrix);
rot_matrix_wld = rotation_matrix.t();
translation_vec_wld = -rot_matrix_wld * translation_vector;
I used the rot_matrix_wld, translation_vec_wld to visualize the pose (same way as how I visualized the pose of the object in the camera coordinate system as said in the above)
projectPoints(points_to_project, rot_matrix_wld, translation_vec_wld, iphone_camera_matrix, iphone_dist_coeffs, projected_points);
with
points_to_project :
[0, 0, 0;
20, 0, 0;
0, 20, 0;
0, 0, 20]
am getting wrong translation vector (below 2 projected_points are for 2 different image frames of a video)
projected_points :
[-795.11768, -975.85846;
-877.84937, -932.39697;
-868.5517, -1197.4443;
projected_points :
[589.42999, 3019.0732;
590.64789, 2665.5835;
479.49728, 2154.8057;
187.78407, 3333.3054]
-593.41058, -851.74432]
I have used the approx camera model and calibrated camera model both are giving the wrong translation vector.
I have gone through the link [here](https://stackoverflow.com/questions/47723638/output-from-solvepnp-doesnt-match-projectpoints) and verified my calibration procedure, I did it correctly.
I am not sure where am doing wrong can anyone please help me with this.
thanks in advance.
slvFri, 29 Jun 2018 03:58:36 -0500http://answers.opencv.org/question/194724/solvePnP for equirectangular imagehttp://answers.opencv.org/question/194389/solvepnp-for-equirectangular-image/ I have an equirectangular image of a room (360), and an accurate model of the room.
I have several matches between image-points and object-points (coordinates in the room) and I want to find the position and orientation of the camera where the image was taken.
SolvePnp (and solvePnPRansac) expect to receive the intrinsic parameters of the camera or to evaluate it, but since the image is equirectangular I can directly know the vector describing each pixel in the image (pitch and yaw).
I managed to solve it manually using some iterative algorithm for least-squares of the offsets between the vectors (something like Gauss-Newton). But it's not very fast and seems like the wrong way to go...YakirSat, 23 Jun 2018 06:30:46 -0500http://answers.opencv.org/question/194389/Clarification of PnP variantshttp://answers.opencv.org/question/193859/clarification-of-pnp-variants/ Hi,
Recently I have been trying out the various PnP algorithms that are part of the openCV library, such as:
1. P3P
2. EPNP
3. ITERATIVE
4. UPNP
5. DLS
After manually annotating a few images for the 2D correspondences (like in [this tutorial](https://docs.opencv.org/3.0-beta/doc/tutorials/calib3d/real_time_pose/real_time_pose.html)), I ran into a few issues and hence
**My question:**
1. UPNP , EPNP and DLS gave me the same result for all images I used. I think that UPNP and EPNP may have same result since UPNP is like EPNP for uncalibrated cameras (while my camera is calibrated), but **I don't understand why DLS is giving me the same result.**
2. Can P3P algorithm accept more than 4 inputs ? Currently openCV throws an assertion error if I pass in >4 points using the P3P flag to the `cv::solvePnP()` method.malharjajooFri, 15 Jun 2018 04:16:47 -0500http://answers.opencv.org/question/193859/c++, calling opencv from multiple threadshttp://answers.opencv.org/question/191753/c-calling-opencv-from-multiple-threads/Hi,
i have a highly parallel problem: run solvePnP() on several thousand independent input datasets.
Using a single c++ pthread to call the lib is only using a small part of the CPU cores available. IPP / TBB / whatever else opencv uses seem to be limited in the number of CPU cores used per dataset for this problem and input datasets.
Using #CPU pthreads is faster overall, but also hits a limit at around 3 threads: something in the lib seems to lock a global critical section.
Is it possible to get around this critical section without running multiple processes? Which compile time options are relevant here?
I'm currently using the OpenCV 3.4.1 Windows Pack, but i'm flexible.
Thanksl.gilsonThu, 17 May 2018 09:52:47 -0500http://answers.opencv.org/question/191753/Multi-view solvePnP routinehttp://answers.opencv.org/question/131660/multi-view-solvepnp-routine/Hi all!
I need a multi-view version of the solvePnP function.
Qualitively:
We want to resolve the pose (rotation + translation) of an object in space using projections of landmarks on that object onto multiple image planes. Each image plane represents a calibrated camera at fixed locations in the world (for each we have a priori : cameraMatrix, distortionCoefficients, rotation, translation). The object is covered in markers (e.g. 10-20 markers, perhaps 4-8 will be in the camera's view at any time) which can be seen and identified in the cameras and are at known 3D positions in object space and known corresponding 2D positions in each image plane. Using the correspondences of 3D points in object space to 2D points in projected image space for each camera, we must reliably (and quickly) discover the rotation and translation for the object.
Quantitively:
* Inputs
* Set[ intrinsics, extrinsics ] views // size N
* Set[ Set[ObjectPoints], Set[ImagePoints] ] // size N
* Outputs
* rotation of object // 3-vector
* translation of object // 3-vector
I have some notes at:
https://paper.dropbox.com/doc/KC35-stereoSolvePnP-pseudo-code-14VMJDF9W8UhMxVOGdCEZ
And posed a freelancer posting at:
https://www.upwork.com/jobs/~01b0f0c4105c0652da
Using a single camera this is possible using the solvePnP function (which optimises the rotation and translation so that the projections of object points match the observed points on the image plane)
Template for the function could be:
double solvePnPMultiView(vector<vector<cv::Point3f>> objectPointsPerView
, vector<vector<cv::Point2f>> imagePointProjectionsPerView
, vector<cv::Mat> cameraMatrixPerView
, vector<cv::Mat> distortionCoefficientsPerView
, vector<cv::Mat> translationPerView
, vector<cv::Mat> rotationVectorPerView
, cv::Mat & objectRotationVector
, cv::Mat & objectTranslation
, bool useExtrinsicGuess);
//same function but with different data format
double solvePnPMultiView(vector<vector<cv::Point3f>> objectPointsPerView
, vector<vector<cv::Point2f>> undsitortedImagePointProjectionsPerView
, vector<cv::Mat> rectifiedProjectionMatrixPerView
, cv::Mat & objectRotationVector
, cv::Mat & objectTranslation
, bool useExtrinsicGuess);
//specific version for stereo (calls one of the functions above)
double solvePnPStereo(vector<cv::Point3f> objectPointsObservedInCamera1
, vector<cv::Point2f> projectedImagePointsObservedInCamera1
, vector<cv::Point3f> objectPointsObservedInCamera2
, vector<cv::Point2f> projectedImagePointsObservedInCamera2
, cv::Mat cameraMatrix1
, cv::Mat distortionCoefficientsCamera1
, cv::Mat cameraMatrix2
, cv::Mat distortionCoefficientsCamera2
, cv::Mat camera1ToCamera2RotationVector
, cv::Mat camera1ToCamera2Translation
, cv::Mat & objectRotationVector
, cv::Mat & objectTranslation
, bool useExtrinsicGuess);
(these functions would all call the same code internally, but just have different ways of being used)
The object we're trying to track is a tree (with known mesh taken from photo-scan). The tree is covered in retroreflective markers. We are projecting onto the tree from a set of moving video projectors (attached to robot arm). I'm pretty confident I can figure out which marker is which before we get to the solvePnP stage. This is all part of a new artwork by our studio (please check http://kimchiandchips.com#lightbarriersecondedition for an example of previous work).
Notes:
* Routine should take less than 3ms on Core i7 for 2 views with 10 object points each.
* Ideally don't use any libraries other than OpenCV (would be even be great to PR this into OpenCV)
* I think OpenCV's only numerical solver is CvLevMarq which is C only, but I'd like to use C++ style where possible (i.e. it seems probably might need to dip into C-only when working with the solver).
* Correctly calculating the derivatives for the solver is essential for reliability and speed
This routine will be employed in an open-source motion capture system. Please see https://github.com/elliotwoods/ofxRulr/tree/MoCap/Plugin_MoCap/src/ofxRulr/Nodes/MoCap for example of code so far (by me)
Thank you
ElliotelliotwoodsTue, 07 Mar 2017 08:01:01 -0600http://answers.opencv.org/question/131660/Strange ArUco behavior / OpenCV SolvePnphttp://answers.opencv.org/question/190853/strange-aruco-behavior-opencv-solvepnp/Hi There,
I tested the accuracy of ArUco markers in severeal distances and used for that a board with a not rotated marker, several rotated marker on their z-Axis (known rotation) and several more not rotated marker. All markers are measured in their rotation in relation to the first not rotated marker.
Now I calculate the transformation (with quaternions) from the marker of interest in the reference marker. The output for my accuracy is the angle theta from the axis angles. The strange thing is, that the rotation error of the rotated (5°, 20°, 30°, 45°, 90° 180°) markers is small (max. 1°) and the error of the not roted marker (0°) is large (>2°).
I do the subpixel corner refinement and the detected markers looks fine for me. I can exclude the error of the camera, because changing positions (near to the center of the image or not) does not change the accuracy. As well I switched the IDs.
How can it be, that rotated marker are more accurate then non roated marker. Could it be a singularity on the detection or numeric errors?
Thank you for your help!
Sarahsarah1802Fri, 04 May 2018 02:31:17 -0500http://answers.opencv.org/question/190853/how to match 3d-2d corresspondence for using solvePnPhttp://answers.opencv.org/question/190524/how-to-match-3d-2d-corresspondence-for-using-solvepnp/ i have problem to match 3d feature from triangulate feature (frame-2,frame-1) and feature2d from current frame to be used in cv2.solvePnP because the feature2d have more feature(in my case 138 after matching with feature from frame-1).
#i triangulate feature frame-2 frame
points4D=cv2.triangulatePoints(proj1,proj2,points11,points22)
points3D= points4D[:3,:]
points3D=points3D/points3D[3]
points3D=np.transpose(points3D)
after that i match feature frame-1 and curr frame and get points33 with 138 feature but the point3D only have 98
points33=np.ascontiguousarray(points3[:,:2].reshape(-1,1,2))
ret,rvec,tvec=cv2.solvePnP(points3D,points3,cMtx,disCo,flags=cv2.SOLVEPNP_DLS)
how to match feature3d(points3D) and feature2d(points33) used so i can use SolvePnP ?
ibiMon, 30 Apr 2018 01:34:20 -0500http://answers.opencv.org/question/190524/SolvePnp results to Quaternion, euler flipping.http://answers.opencv.org/question/186967/solvepnp-results-to-quaternion-euler-flipping/I am using aruco markers and solvePnp to return A camera pose. I run PnP, then I use the following function to get the camera pose as a quaternion rotation from the rvec and tvec:
void GetCameraPoseEigen(cv::Vec3d tvecV, cv::Vec3d rvecV, Eigen::Vector3d &Translate, Eigen::Quaterniond &quats)
{
Mat R;
Mat tvec, rvec;
tvec = DoubleMatFromVec3b(tvecV);
rvec = DoubleMatFromVec3b(rvecV);
cv::Rodrigues(rvec, R); // R is 3x3
R = R.t(); // rotation of inverse
tvec = -R*tvec; // translation of inverse
Eigen::Matrix3d mat;
cv2eigen(R, mat);
Eigen::Quaterniond EigenQuat(mat);
quats = EigenQuat;
double x_t = tvec.at<double>(0, 0);
double y_t = tvec.at<double>(1, 0);
double z_t = tvec.at<double>(2, 0);
Translate.x() = x_t * 10;
Translate.y() = y_t * 10;
Translate.z() = z_t * 10;
}
This works, but i am still seeing huge flipping in the rotation values at some angles. My pose flips 180 degrees, and then back.
I am running the DrawAxis to check the pose in my camera frame, and this looks solid, yet my converted values do not. Where am i going wrong? Is there a mistake in the above function?antithingSat, 17 Mar 2018 08:40:38 -0500http://answers.opencv.org/question/186967/solvePnP fluctuating resulthttp://answers.opencv.org/question/186946/solvepnp-fluctuating-result/Hi, I am using solvePnp to detect head pose. But estimate result from rotation, translation fluctuating a lots.
solvePnp Call:
cv::solvePnP(model3DPoints, imagePoints, cameraMatrix, dist_coeffs, rotationMat, translationMat);
Camera matrix code:
float focal_length = frame.cols;
cv::Point2d center = cv::Point2d(frame.cols / 2, frame.rows / 2);
cameraMatrix = (cv::Mat_<double>(3,3) << focal_length, 0, center.x, 0 , focal_length, center.y, 0, 0, 1);
Camera matrix value in real time:
640.000000 0.000000 320.000000
0.000000 640.000000 426.000000
0.000000 0.000000 1.000000
Distortion coefficients matrix:
dist_coeffs = cv::Mat::zeros(4,1,cv::DataType<double>::type);
3D Models Points:
model3DPoints.push_back(cv::Point3f(6.825897, 6.760612, 4.402142));
model3DPoints.push_back(cv::Point3f(1.330353, 7.122144, 6.903745));
model3DPoints.push_back(cv::Point3f(-1.330353, 7.122144, 6.903745));
model3DPoints.push_back(cv::Point3f(-6.825897, 6.760612, 4.402142));
model3DPoints.push_back(cv::Point3f(5.311432, 5.485328, 3.987654));
model3DPoints.push_back(cv::Point3f(1.789930, 5.393625, 4.413414));
model3DPoints.push_back(cv::Point3f(-1.789930, 5.393625, 4.413414));
model3DPoints.push_back(cv::Point3f(-5.311432, 5.485328, 3.987654));
model3DPoints.push_back(cv::Point3f(2.005628, 1.409845, 6.165652));
model3DPoints.push_back(cv::Point3f(-2.005628, 1.409845, 6.165652));
model3DPoints.push_back(cv::Point3f(2.774015, -2.080775, 5.048531));
model3DPoints.push_back(cv::Point3f(-2.774015, -2.080775, 5.048531));
model3DPoints.push_back(cv::Point3f(0.000000, -3.116408, 6.097667));
model3DPoints.push_back(cv::Point3f(0.000000, -7.415691, 4.070434));
2D Points value:
(156.139725, 367.403290)
(288.635376, 340.711731)
(358.303589, 343.156128)
(495.513824, 371.169189)
(199.867783, 404.109467)
(272.105225, 411.756317)
(379.791809, 412.740295)
(453.119293, 409.567810)
(276.528778, 520.536621)
(371.074890, 519.674438)
(241.585831, 596.283813)
(409.880096, 592.829407)
(327.109375, 641.688782)
(330.348145, 716.255615)
Only the 2d image point change a bit but the result change a lots.
ROTATION (-3452928277881874609248275663945728.000000, -0.915058, 0.000000)
TRANSLATION (-3635378688.000000, -1.656431, -385174336.000000)
ROTATION (0.000000, -0.899182, 0.005109)
TRANSLATION (0.000000, -1.649161, 1274852533763833856.000000)
ROTATION (-0.000000, -0.953833, 7.130205)
TRANSLATION (-0.000000, -1.669374, -0.001339)
ROTATION (0.000052, 0.986896, 1245620912259072.000000)
TRANSLATION (-0.000000, -1.616727, 0.000000)Luan Tran MinhSat, 17 Mar 2018 04:50:03 -0500http://answers.opencv.org/question/186946/OpenCv SolvePnp result, to Eigen Quaternionhttp://answers.opencv.org/question/182862/opencv-solvepnp-result-to-eigen-quaternion/I am running opencv solvePnP using 3d points that I am passing in manually, and 2d image points from the corner of an aruco marker.
my 4 x 2d, and 4 x 3d points are as follows:
2d: [631.811, 359.303]
2d: [750.397, 364.894]
2d: [761.65, 448.106]
2d: [621.837, 440.352]
3d: [2.73788, -2.19532, 119.784]
3d: [-18.8274, -3.25251, 119.167]
3d: [-17.7176, -15.5556, 101.448]
3d: [3.84767, -14.4984, 102.065]
my camera matrix is:
[657.0030186215286, 0, 646.6970647943629;
0, 657.0030186215286, 347.1175117491306;
0, 0, 1]
When I run this, I get rvec and tvec:
rot: [3.542907682527767e-08;
9.321253763995684e-10;
3.141592612624269]
tran: [-2.599387317758439e-06;
7.428777431217353e-07;
-1.306287668720693e-06]
Already, this looks strange. Please see the image attached for an example of where the marker is in relation to the camera. Then, i run the rvec and tvec through a function to get the camera pose in relation to the marker as an `Eigen::quaterniond`
void GetCameraPoseEigen(cv::Vec3d tvecV, cv::Vec3d rvecV, Eigen::Vector3d &Translate, Eigen::Quaterniond &quats)
{
Mat R;
Mat tvec, rvec;
tvec = DoubleMatFromVec3b(tvecV);
rvec = DoubleMatFromVec3b(rvecV);
cv::Rodrigues(rvec, R); // R is 3x3
R = R.t(); // rotation of inverse
tvec = -R*tvec; // translation of inverse
Eigen::Matrix3d mat;
cv2eigen(R, mat);
Eigen::Quaterniond EigenQuat(mat);
quats = EigenQuat;
double x_t = tvec.at<double>(0, 0);
double y_t = tvec.at<double>(1, 0);
double z_t = tvec.at<double>(2, 0);
Translate.x() = x_t * 10;
Translate.y() = y_t * 10;
Translate.z() = z_t * 10;
}
The resulting quaternion is:
`1.12774e-08 2.96705e-10 1 -2.04828e-08`
Which is obviously wrong. But where is my mistake? Do my 2d - 3d correspondences look correct? Or is it my Eigen conversion?
Thank you.
![image description](/upfiles/15164543231226668.png)
EDIT:
I have adjusted the getCorners function based on the helpful comments below.
std::vector<cv::Point3f> getCornersInWorld(double side, cv::Vec3d rvec, cv::Vec3d tvec) {
//marker half size
double half_side = side / 2;
// compute rot_mat
cv::Mat rot_mat;
cv::Rodrigues(rvec, rot_mat);
// transpose of rot_mat
cv::Mat rot_mat_t = rot_mat.t();
//points in marker space
cv::Point3d cnr1(0, -half_side, half_side);
cv::Point3d cnr2(0, half_side, half_side);
cv::Point3d cnr3(0, half_side, -half_side);
cv::Point3d cnr4(0, -half_side, -half_side);
//to mat
cv::Mat cnr1Mat(cnr1);
cv::Mat cnr2Mat(cnr2);
cv::Mat cnr3Mat(cnr3);
cv::Mat cnr4Mat(cnr4);
//rotate points
cv::Mat pt_newMat1 = (-rot_mat_t * cnr1Mat);
cv::Mat pt_newMat2 = (-rot_mat_t * cnr2Mat);
cv::Mat pt_newMat3 = (-rot_mat_t * cnr3Mat);
cv::Mat pt_newMat4 = (-rot_mat_t * cnr4Mat);
// convert tvec to point
cv::Point3d tvec_3d(tvec[0], tvec[1], tvec[2]);
cv::Mat tvecMat(tvec_3d);
//rotate tvec
cv::Mat tvec_new = (-rot_mat_t * tvecMat);
cv::Point3d p(tvec_new);
//subtract tvec from all
pt_newMat1 = pt_newMat1 - tvec_new;
pt_newMat2 = pt_newMat2 - tvec_new;
pt_newMat3 = pt_newMat3 - tvec_new;
pt_newMat4 = pt_newMat4 - tvec_new;
//back to pnt
cv::Point3d p1(pt_newMat1);
cv::Point3d p2(pt_newMat2);
cv::Point3d p3(pt_newMat3);
cv::Point3d p4(pt_newMat4);
std::vector<cv::Point3f> retPnts;
retPnts.push_back(p1);
retPnts.push_back(p2);
retPnts.push_back(p3);
retPnts.push_back(p4);
return retPnts;
}
This gives me:
2d: [640.669, 489.541]
2d: [746.65, 479.212]
2d: [787.032, 547.899]
2d: [662.316, 561.966]
3d: [26.7971, 83.641, -78.1216]
3d: [28.0609, 91.2866, -57.9595]
3d: [31.636, 111.129, -65.7082]
3d: [30.3722, 103.484, -85.8702]
rot: [-0.4156437942693543;
1.047535551778523;
-1.063860719199172]
tran: [20.6613956945096;
-2.802643913661057;
239.6085645311116]
quat:
-0.187639 0.472901 -0.480271 0.71449
Which looks more sensible, but is still wrong. the resulting pose seems to be around 45 degrees off what I would expect, and on the wrong axis.antithingSat, 20 Jan 2018 07:19:41 -0600http://answers.opencv.org/question/182862/