OpenCV Q&A Forum - RSS feedhttp://answers.opencv.org/questions/OpenCV answersenCopyright <a href="http://www.opencv.org">OpenCV foundation</a>, 2012-2018.Mon, 14 Dec 2020 19:41:25 -0600How to find surfance orientation (relative to camera) from single depth image?http://answers.opencv.org/question/239261/how-to-find-surfance-orientation-relative-to-camera-from-single-depth-image/ Supposing I have access to the image stream of a depth camera, and there is a flat surface (e.g. floor, tabletop, etc) within the camera's FoV at all times, how could one estimate the floor vertical and horizontal orientation (or better yet, the rotation matrix/vector) from the camera perspective?
I have access to the camera matrix, therefore I can select multiple points on the surface and reconstruct their 3D coordinates on the camera frame. But how do I use those coordinates to build a transformation matrix (mostly rotation, translation is irrelevant, I'd just need to orient my reference frame orthogonally with the surface)
My main limitation seems to be that I do not have the correspondent coordinates on the surface points (on object/external reference), therefore can't use cv::`estimateAffine3D` or `cv::findHomography` or `cv::solvePnP`.
I've tried to estimate the plane equation using cv::SVD, but resulting fit doesn't seem to be that precise and I am not aware how could I use the plane equation to find the affine transformation matrix. joaocandreMon, 14 Dec 2020 19:41:25 -0600http://answers.opencv.org/question/239261/Pose estimation of image from a point cloudhttp://answers.opencv.org/question/238529/pose-estimation-of-image-from-a-point-cloud/I have a 3D point cloud of scene, its set of views with known poses and a new query image from camera with camera calibration data (i.e. focal length). Here is a snapshot of my point cloud of scene
![image description](/upfiles/16068386634272687.png)
and query image
![image description](/upfiles/16068386779254414.jpg)
I need to estimate the camera pose of the query image.
I have found similar [question](https://stackoverflow.com/questions/44445851/how-to-estimate-camera-pose-matrix-of-a-new-image-i-from-a-known-3d-point-cloud) on SO. My intuition is to find point correspondences btw the query image and all the view image of the scene and apply SolvePnP to get the camera pose.
My questions is whether it is right solution and how to approach it? Should I somehow use the original point cloud here once I get good correspondences btw the query image and view images to estimate the camera pose?sigmoid90Tue, 01 Dec 2020 10:18:57 -0600http://answers.opencv.org/question/238529/Projecting point cloud to image plane in order to estimate camera posehttp://answers.opencv.org/question/238349/projecting-point-cloud-to-image-plane-in-order-to-estimate-camera-pose/I am working on a task of image registration in point cloud of scene. I have an image from camera, 3D point cloud of scene and camera calibration data (i.e. focal length). The image and point cloud of scene share the same space.
Here is a snapshot of my point cloud of scene
![image description](/upfiles/16067296657068877.png)
and query image
![image description](/upfiles/16067296871448465.jpg)
From my previous question [here](https://answers.opencv.org/question/238269/error-assertion-failed-scn-3-scn-4-in-cvtcolor-when-calculating-orb-features-on-point-cloud/) I have learnt that the SolvePnP method from OpenCV would work in this case but there is no obvious way to find robust correspondences btw query image and scene point cloud.
My intuition now is to project point cloud to image plane, match keypoints calculated on it against ones calculated on query image and use robust point correspondences in the SolvePnP method to get camera pose.
I have found [here](https://stackoverflow.com/questions/40677116/conversion-of-cloud-data-into-2d-image-using-opencv) a way to project point cloud to image plane.
My questions is could it work and is it possible to preserve the transformation btw the resultant image of scene and the original point cloud?sigmoid90Fri, 27 Nov 2020 05:36:50 -0600http://answers.opencv.org/question/238349/Technique to find corresponding objects across stereo viewshttp://answers.opencv.org/question/238251/technique-to-find-corresponding-objects-across-stereo-views/Hi all,
Thanks for taking your time to read this.
We have fixed stereo pairs of cameras looking into a closed volume. We know the dimensions of the volume and have the intrinsic and extrinsic calibration values
for the camera pairs. The objective being to be able to identify the 3d positions of multiple duplicate objects accurately.
Which naturally leads to what is described as the correspondence problem in litrature. We need a fast technique to match ball A from image 1 with Ball A from image 2 and so on.
At the moment we use the properties of epipolar geomentry (Fundamental matrix) to match the balls from different views in a crude way and works ok when the objects are sparse,
but gives a lot of false positives if the objects are densely scattered. Since ball A in image 1 can lie anywhere on the epipolar line going across image 2, it leads to mismatches
when multiple objects lie on that line and look similar.
Is there a way to re-model this into a 3d line intersection problem or something? Since the ball A in image a can only take a bounded limit of 3d values, Is there a way to represent
it as a line in 3d? and do a intersection test to find the closest matching ball in image 2?
Or is there a way to generate a sparse list of 3d values which correspond to each 2d grid of pixels in image 1 and 2, and do a intersection test
of these values to find the matching objects across two cameras?
Because the objects can be identical, OpenCV feature matching algorithms like FLANN, ORB doesn't work.
Any ideas in the form of formulae or code is welcome.
Thanks!
Sak
![image description](/upfiles/16063035094014803.png)
![image description](/upfiles/16063035277440008.png)SakWed, 25 Nov 2020 05:27:22 -0600http://answers.opencv.org/question/238251/OpenCV solvePnPRansac() functionhttp://answers.opencv.org/question/232035/opencv-solvepnpransac-function/ I am working on a 3D model reconstruction form multiple images project.
I have the camera matrix as well as 2D-3D point correspondence. I want to compute the projection matrix.
I used <code>cv.solvePnPRansac()</code>function to get the projection matrix as follow:
retval, rvec, tvec, inliers = cv.solvePnPRansac(np.ascontiguousarray(np.squeeze(np.array(selectedpts3D)) [:,:-1]).reshape(len(selectedpts3D),1,3), np.ascontiguousarray(np.array(pts)[:,:-1]).reshape(len(pts),1,2), np.array(K_global), np.array([]), flags = cv.SOLVEPNP_ITERATIVE)
rvec_matrix = cv.Rodrigues(rvec)[0]
proj_matrix = np.hstack((rvec_matrix, tvec))
Then I factorize the output projection matrix to get camera matrix, rotation matrix and translation matrix as follow:
def factorize_proj_matrix(P):
""" Factorize the camera matrix into K,R,t as P = K[R|t]. """
# factor first 3*3 part
K1,R = linalg.rq(P[:,:3])
# make diagonal of K positive
T = np.diag(np.sign(np.diag(K1)))
if np.linalg.det(T) < 0:
T[1,1] *= -1
K1 = np.dot(K1,T)
R = np.dot(T,R) # T is its own inverse
t = np.dot(np.linalg.inv(K1),P[:,3])
return K1, R, t.reshape(3,1)
but the resulted camera matrix "K" has different values than the passed camera matrix to solvePnPRansac() function.
passed camera matrix to function is:
[[1.5204e+03 0.0000e+00 3.0232e+02]
[0.0000e+00 1.5259e+03 2.4687e+02]
[0.0000e+00 0.0000e+00 1.0000e+00]]
output camera matrix is:
array([[ 2.43791400e-03, -7.07610743e-21, 3.23838460e-22],
[ 0.00000000e+00, 2.43791400e-03, -2.88994138e-22],
[ 0.00000000e+00, 0.00000000e+00, 2.43791400e-03]])
My question is does <code>cv.solvePnPRansac()</code> function changes the cameraMatrix. if yes how to enforce it not to change the camera matrix?
Sara AhmedFri, 03 Jul 2020 01:13:57 -0500http://answers.opencv.org/question/232035/Coordinate frame transformation from NED to OpenCV convention, and the Pinhole Camera Modelhttp://answers.opencv.org/question/233204/coordinate-frame-transformation-from-ned-to-opencv-convention-and-the-pinhole-camera-model/ Hello,
I'm working on an SFM module where I have camera coordinates wrt a world coordinate system (in NED). I have the calibrated camera intrinsics matrix, and with the R,T of the camera wrt world, I compute the Projection matrix (intrinscs matrix * R|T) for each image, and camera position. However, I'm not sure if this Projection matrix is right since the R|T is in NED, and openCV uses a different convention.I'm trying to use the CV::SFM::triangulatePoints, but I'm not getting consistent results.
How will this affect the projection matrix, and is the conversion of the poses to the OpenCV frame necessary?
leafdetSun, 02 Aug 2020 12:28:00 -0500http://answers.opencv.org/question/233204/Connection between pose estimation, epipolar geometry and depth maphttp://answers.opencv.org/question/233007/connection-between-pose-estimation-epipolar-geometry-and-depth-map/ Hi I am an undergraduate student working on a graduate project, and a beginner to computer vision.
After I went through the tutorial "Camera Calibration and 3D Reconstruction" provided by OpenCV (link) :
https://docs.opencv.org/master/d9/db7/tutorial_py_table_of_contents_calib3d.html
I failed to see the connections between the second part to the final part. What I understand here is :
- The intrinsic and extrinsic parameters of a camera is required to estimate the position of the camera and the
captured object
- To reconstruct a 3D model multiple point clouds are needed, and to generate a point cloud a disparity map is required.
What I do not understand is :
- The importance of estimating the position of the camera or the object to compute the epiline or epipole in either image planes.
- The importance of epipolar geometry, and finding out the location of epiline and epipole to compute the disparity map.
As far as I am aware, the code below generate a disparity map
stereo = cv2.createStereoBM(numDisparities=16, blockSize=15)
disparity = stereo.compute(imgL,imgR)
and the input includes a pair of stereo images, minDisparities, numDisparities and blockSize, but not the position of the camera nor the epiline/epipole.
Any help would be greatly appreciated.
askl1278Tue, 28 Jul 2020 09:04:27 -0500http://answers.opencv.org/question/233007/OpenCV Decompose projection matrix outputshttp://answers.opencv.org/question/232009/opencv-decompose-projection-matrix-outputs/0
I got confused with the outputs of opencv `decomposeProjectionMatrix` function.
I have the projection matrix and the camera matrix "K" and I want to get the translation vector "t"and the Rotation matrix "R" from the projection matrix
As I know the projection matrix of dimension 3*4 = K[R|t] in which "t" is a 3*1 vector
`cv2.decomposeProjectionMatrix` returns R with dimension 3*3 which is correct but the transVect returned is of dimension 4*1 not 3*1
My question is how to get back the projection matrix from the function outputs?Sara AhmedThu, 02 Jul 2020 12:56:21 -0500http://answers.opencv.org/question/232009/How to calculate disparity image for Left to Right from Right to Left andvice versa?http://answers.opencv.org/question/231963/how-to-calculate-disparity-image-for-left-to-right-from-right-to-left-andvice-versa/In account of reducing the resource usage. i need to derive the Left to Right aggregated cost from Right to Left aggregated cost for consistency check. Any possibility for that?.prasanna_18Thu, 02 Jul 2020 06:53:03 -0500http://answers.opencv.org/question/231963/Best way to gather all pixels location (XY) inside a closed contour?http://answers.opencv.org/question/231309/best-way-to-gather-all-pixels-location-xy-inside-a-closed-contour/Currently i have a black&white 3D model sliced in 2D images (+/- 500 to 1000 images), each layer is 0.05mm in Z, that aside i need to capture all closed black areas contours which i already have. I need to detect all empty spaces on the model looping all images and go up and down in Z at each countour area to find if is hollow space in a 3D prespective or if have any gap to outside the model and will not consider that area hollow.
Currently i'm doing the following:
1. for first layer to last layer:
2. Detect the required contours
3. Foreach contour
4. Create a empty image at same size of input
5. FillConvexPoly at empty image
6. Loop image pixels and find that previous draw pixels and save them in a array
That's the first part of the problem, what i'm doing is very inefficient/slow,... As can be observed each contour in layer image must loop all pixels in image, so if i have 5 countours on a image, it loop image 5 times to get pixels as a countour area.
Current code:
for (int i = 0; i < contours.Size; i++)
{
if ((int)arr.GetValue(0, i, 2) != -1 || (int)arr.GetValue(0, i, 3) == -1) continue;
grayscale.Dispose();
grayscale = new Image<Gray, byte>(image.Width, image.Height);
//grayscale = grayscale.CopyBlank();
grayscale.FillConvexPoly(contours[i].ToArray(), new Gray(125));
List<Point> points = new List<Point>();
byte[,,] data = grayscale.Data;
for (int y = 0; y < grayscale.Rows; y++)
{
for (int x = 0; x < grayscale.Cols; x++)
{
if(data[y, x, 0] != 125) continue;
points.Add(new Point(x, y)); // Gather pixels
}
}
listHollowArea.Add(new LayerHollowArea(points.ToArray()));
}
The question: Is there any function or code i can use to skip 4 to 6 and get all pixels inside the matched countours without that much effort?sn4k3Tue, 16 Jun 2020 23:01:53 -0500http://answers.opencv.org/question/231309/Triangulation scaling problem. (Multiple Images)http://answers.opencv.org/question/231247/triangulation-scaling-problem-multiple-images/Hello everyone,
I try to do 3d metric reconstruction with 60 images.
I use python and there is only [this](https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html#gad3fc9a0c82b08df034234979960b778c) function in opencv to triangulation with python. It reconstructs the images two by two.
There are 2 methods that I consider using to make triangulation.
The first method is to select 1 image as base and triangulate one by one with its neighbors ;
Like this: ![image description](/upfiles/15922198763035918.png)
where 1 -> Base Image
; 2,3,4,5,6 -> Base Image's neighbors
R1 =[[1 0 0] t1 = [[0]]
[0 1 0] [0]
[0 0 1]] [0]]
R2,t2 = cv2.findEssentialMat(img1.pts, img2.pts, K) and cv2.recoverPose(E)
R3,t3 = cv2.findEssentialMat(img1.pts, img3.pts, K) and cv2.recoverPose(E)
.
.
R6,t6 = cv2.findEssentialMat(img1.pts, img6.pts, K) and cv2.recoverPose(E)
P1 = K[R1|t1]
P2 = K[R2|t2]
.
.
P6 = K[R6|t6]
3d_first = cv2.triangulatePoints(P1,P2, pts1,pts2)
3d_second = cv2.triangulatePoints(P1,P3, pts1,pts3)
3d_third = cv2.triangulatePoints(P1,P4, pts1,pts4)
3d_fourth = cv2.triangulatePoints(P1,P5, pts1,pts5)
3d_fifth = cv2.triangulatePoints(P1,P6, pts1,pts6)
When I visualize the 3d points, they do not overlap and there is a difference between them.(I guess It is a scaling problem).
How can i fix the problem?
There is a suggestion like that ;
t1 = t1 * (camera1 and camera2 baseline (unit metric))
t2 = t2 * (camera1 and camera3 baseline (unit metric))
but this sometimes does not work. Does this give me a metric results?
The second method is to triangulate the photos as img1-img2, img2-img3, img3-img4 instead of img1-img2, img1-img3, img1-img4. In this case, I have to bring all the 3D points to the same plane. Because P1 = [I | 0] for img1-img2 triangulation, P2 = [I, 0] for img2-img3 triangulation.
In this case, point clouds for img1-img2 will appear on the plane img1, and point clouds for img2, img3 will appear on the plane 2.
How can I bring them to the same plane?
Thanks in advance.guknaMon, 15 Jun 2020 06:14:23 -0500http://answers.opencv.org/question/231247/Trying to find a solution for real time body dimension recognition, cloth modeling, and on-body simulationhttp://answers.opencv.org/question/229798/trying-to-find-a-solution-for-real-time-body-dimension-recognition-cloth-modeling-and-on-body-simulation/As I understand the problem has 3 major parts.
1. measure human body dimensions
2. create cloths 3d models with markers
3. render 3d model on human body from video stream
NHSat, 02 May 2020 22:57:36 -0500http://answers.opencv.org/question/229798/triangulatePoints and undistortPointshttp://answers.opencv.org/question/229148/triangulatepoints-and-undistortpoints/I need some help in understanding these two functions. So afaik, triangulatePoints takes in a pair of 2D Pixel coordinates from calibrated images, and returns the triangulated point in homogeneous coordinates. I know in general what homogenous coordinates are but I'm confused about the 4th element in the output, what does this scale factor refer to? After we divide all the coordinates by this last element this is supposed to be in euclidean coordinates, but what is the origin for this euclidean coordinate system? Is it the left camera which is usually taken as the origin in stereoCalibrate?
Also, what are the units of the coordinate system, is it the size of the checkerboard square?
In addition, I also do not understand why we need to undistortPoints and what does undistortPoints exactly give us. I am hoping to find all my answers here because I am unable to understand properly anything from similar questions asked on stack overflow regarding these functions and the documentation isn't very informative PyNinjaFri, 17 Apr 2020 13:39:17 -0500http://answers.opencv.org/question/229148/Procedure for getting 3D coordinates from Stereo?http://answers.opencv.org/question/228743/procedure-for-getting-3d-coordinates-from-stereo/This question has been asked in different ways many times, many people approach this problem in different ways. Which is why I am unsure of my methodology, it would be good if someone could tell me if I am doing things correctly.
The steps that I follow are -
- Set up stereo camera pair.
- Calibrate both the cameras individually to get
their camera matrices, distortion
coefficients.
- Then we calculate R, T,
E, F using `stereoCalibrate`
- Calculate
R1,R2,P1,P2 and Q using `stereoRectify`
- Since I want to find the 3D coordinates of a specific point in my scene, I extract the coordinates of that point in both the images, and after that I use `triangulatePoints` to get the 3D points in homogenous coordinates.
Are the steps that I'm following correct?
PyNinjaWed, 08 Apr 2020 17:59:26 -0500http://answers.opencv.org/question/228743/Auto rotation of two point clouds from stereo righttp://answers.opencv.org/question/225537/auto-rotation-of-two-point-clouds-from-stereo-rig/I built this stereo camera rig and eventually I was able to get to the point where I could extract pretty decent point clouds from it. The point cloud comes from: cv2.reprojectImageTo3D(disparity, disparity_to_depth_map) and when I merged it with color data it looks like a faithful 3D representation of my desk when I view it. Now I want to learn how to stitch multiple point clouds together automatically. Basically I want to take pictures from multiple angles and stitch them together. From reading it seems that if I can get them close enough then I can use ICP on them to merge them together. I know that I could do the initial setup manually but I want to try to do it with my own code.
So I was thinking that I could calculate the optical flow of some feature points between the first set of stereo images and the second. Then since I know the depth at each point I could use solvePNP to get the pose of the first and second pictures. And then I could just subtract the rotation and translation vectors to understand how to rotate my point cloud. Does this make sensor or I am missing something here?
I also added an IMU to my camera so I could try to track movement that way, but I think it would drift too much.
Thanks!eric_engineerThu, 30 Jan 2020 13:06:55 -0600http://answers.opencv.org/question/225537/Single shot structured lighthttp://answers.opencv.org/question/225175/single-shot-structured-light/How this method can be made with openCV.![image description](/upfiles/15796890801453933.png)TonifdsWed, 22 Jan 2020 04:31:49 -0600http://answers.opencv.org/question/225175/3d reconstruction of highly reflective surfacehttp://answers.opencv.org/question/224839/3d-reconstruction-of-highly-reflective-surface/ Any method is suitable for 3D reconstruction of metal surface without coating it with opaque powder.TonifdsTue, 14 Jan 2020 13:33:15 -0600http://answers.opencv.org/question/224839/How do I create a 3D map of keypoints from a single camera?http://answers.opencv.org/question/223909/how-do-i-create-a-3d-map-of-keypoints-from-a-single-camera/I am working on an object tracking solution that needs to track an object in a scene. I have done research and found that i need to create a 3D map of the keypoints in the scene.
So far, i have found a way to collect the keypoints and find the distance from the camera to the point, but how do i use multiple sets of keypoints to map the object in 3D space (like vuforia does).
The problem rises when i need to see how much the camera has moved (the angle by which it has moved and its positional movement).
How is this solved? Or better yet, can you refer me to a well-documented SLAM code which i can take as a reference in developing my own SLAM based tracking.Dimitar TBTue, 24 Dec 2019 07:27:20 -0600http://answers.opencv.org/question/223909/stereoBM and SGBM not workinghttp://answers.opencv.org/question/220586/stereobm-and-sgbm-not-working/I am trying to test the OpenCV classes stereoBM and StereoSGBM classes. when compiling the code below using
>
g++ main.cpp -o app `pkg-config --cflags --libs opencv`
I get this error,
> fatal error: opencv2/contrib/contrib.hpp: No such file or directory
I am using Ubuntu 18.04 on an ARM platform (ODROID-XU4), and I have installed OpenCV version 3.2 using Synaptic Package Manager. The libraries are installed as indicated in the picture. However, I cannot locate contrib.hpp
![image description](/upfiles/15722878755557377.png)
#include "opencv2/core/core.hpp"
#include "opencv2/calib3d/calib3d.hpp"
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include "opencv2/contrib/contrib.hpp"
#include <stdio.h>
#include <string.h>
using namespace cv;
using namespace std;
int main(int argc, char* argv[])
{
Mat g1, g2;
Mat disp, disp8;
char* method = argv[3];
g1 = imread(argv[1]);
g2 = imread(argv[2]);
//cvtColor(img1, g1, CV_BGR2GRAY);
//cvtColor(img2, g2, CV_BGR2GRAY);
if (!(strcmp(method, "BM")))
{
StereoBM sbm;
sbm.state->SADWindowSize = 9;
sbm.state->numberOfDisparities = 112;
sbm.state->preFilterSize = 5;
sbm.state->preFilterCap = 61;
sbm.state->minDisparity = -39;
sbm.state->textureThreshold = 507;
sbm.state->uniquenessRatio = 0;
sbm.state->speckleWindowSize = 0;
sbm.state->speckleRange = 8;
sbm.state->disp12MaxDiff = 1;
sbm(g1, g2, disp);
}
else if (!(strcmp(method, "SGBM")))
{
StereoSGBM sbm;
sbm.SADWindowSize = 3;
sbm.numberOfDisparities = 144;
sbm.preFilterCap = 63;
sbm.minDisparity = -39;
sbm.uniquenessRatio = 10;
sbm.speckleWindowSize = 100;
sbm.speckleRange = 32;
sbm.disp12MaxDiff = 1;
sbm.fullDP = false;
sbm.P1 = 216;
sbm.P2 = 864;
sbm(g1, g2, disp);
}
normalize(disp, disp8, 0, 255, CV_MINMAX, CV_8U);
imshow("left", g1;
imshow("right", g2);
imshow("disp", disp8);
waitKey(0);
return(0);
}
TakiEddineMon, 28 Oct 2019 14:31:55 -0500http://answers.opencv.org/question/220586/Can Letters or strings be used as a feature that used to track an object?http://answers.opencv.org/question/220216/can-letters-or-strings-be-used-as-a-feature-that-used-to-track-an-object/ I am creating a tracking program that gives the real-time position of a feature, in this case, a string of characters, that would be attached to helmets of people. I am using 3D Reconstruction functions to determine the coordinates of the feature. luckytiger_21Tue, 22 Oct 2019 09:11:37 -0500http://answers.opencv.org/question/220216/Get Rotation and Translation Matrixhttp://answers.opencv.org/question/217954/get-rotation-and-translation-matrix/I'm programming one Asus Xtion depth camera, wich instead of an RGB image, it gives me the Depth information.
I already have the Camera and Distortion Matrices of the depth camera, but now, I want to calibrate the vision system, by getting the rotation and translation matrices.
I already have the 3D local coordinates of the points, from the camera perspective, but now I need to convert them to world/global coordinates. Since this camera has only depth information I was thinking: is it possible to calibrate this vision system by saying where is the ground plane? How shoul I proceed to put the blue plane as the ground plane of my vison system?
![image description](/upfiles/15676982772248991.jpg)
(note, in addition to the ground plane there's also an object on the plane)
I already tried using the solvePnP to get the rotation and translation matrices, but with no luck. Thanks in advance.dbots94Thu, 05 Sep 2019 10:47:25 -0500http://answers.opencv.org/question/217954/triangulation points with projection matrix in stereorectifyhttp://answers.opencv.org/question/217926/triangulation-points-with-projection-matrix-in-stereorectify/ Hello,
I have two cameras with known internal calibration and exterior orientation.
After using the stereoRectify function, I want to triangulate the points in stereo geometry using the projection matrices P1 and P2, but the 3D coordinate is in relation to the local system created from the left camera, centered at (0,0,0). How do I get the coordinate in my world system?correiaWed, 04 Sep 2019 17:09:07 -0500http://answers.opencv.org/question/217926/triangulatePoints about 3D Reconstructionhttp://answers.opencv.org/question/212640/triangulatepoints-about-3d-reconstruction/I'm using triangulation to proceed with 3D reconstruction.
so I used one 2D camera and I stored the reference image.(intrinsic parameter is same)
reference Image: R1, t1
cam: R2, t2
solvePnP(...,rv1,t1)
rodrigues(rv1, R1)
solvePnP(...,rv2,t2)
rodrigues(rv2, R2)
so
The relative pixel location between the camera can be expressed as:
C0 = R1 * W + t1
C1 = R2 * W + t2
W = R1.inv() * (C0 - t1)
C1 = R2(R1.inv() * (C0 - t1) + t2
R = R2 * R1.inv()
T = t2 -R2 * R1.inv() * t1
and then
cv::Mat P0 = K * cv::Mat::eye(3, 4, CV_64F); // reference image
cv::Mat Rt, X;
cv::hconcat(R, T, Rt);
cv::Mat P1 = K * Rt; // live cam
cv::triangulatePoints(P0, P1, points0, points1, X); (points: 2D image Coordinate using SIFT matching)
it was not an answer.
But if P0 = (R1, t1) and P1 = (R2, t2), it worked well.
Why did not the expression using the relative relationship between the cameras work?
Did I make a mistake?jjjaTue, 07 May 2019 02:51:36 -0500http://answers.opencv.org/question/212640/Does cv::triangulatePoints refine results whe used with several points ?http://answers.opencv.org/question/212160/does-cvtriangulatepoints-refine-results-whe-used-with-several-points/ Hello,
I'm currently working on 3D reconstruction and I'm trying to compare different methods to find 3D coordonates of points from a pair of camera.
I used the cv::triangulatePoints function, which can take an array of 2D points and return the array of corresponding 3D coordinates. I was wondering if this function performs any refinement when used with several points ? Explained in another way, does calling the function once with the n points or n times with each point separately will produce the same results (except in ressources) ?
Thank you for your help.ArchjbaldFri, 26 Apr 2019 09:16:24 -0500http://answers.opencv.org/question/212160/opencv4android for 3d reconstructionhttp://answers.opencv.org/question/211435/opencv4android-for-3d-reconstruction/ Hi, i currently develop an android application regard 3d reconstruction from list of images by using Java. I'm quite confuse to the proper way for the 3d reconstruction steps to be done. Can anyone guide me to the correct path from detect keypoints from images until get a proper 3d point cloud for the 3d model rendering? Sorry for my poor English. ThanksManyToLearnThu, 11 Apr 2019 12:05:47 -0500http://answers.opencv.org/question/211435/Struggling with reprojectImageTo3dhttp://answers.opencv.org/question/210683/struggling-with-reprojectimageto3d/Hi, everyone. I'm trying to triangulate some points lying on a plane in a setup which involves two cameras.
![Reference image](https://imgur.com/gOps4vP) [link text](https://imgur.com/gOps4vP)
![The other image](https://imgur.com/VIiH9Rv) [link text](https://imgur.com/VIiH9Rv)
First of all, I solve the relative pose problem using the 5pts algorithm on the undistorted points for the Essential Matrix estimation, the I recover the pose. I'm using RANSAC.
Then, I rectify the stereo pairs the usual way:
R1, R2, Pn1, Pn2, Q, _, _ = cv2.stereoRectify(K1, dcoeffs1, K2, dcoeffs2,
img1.shape[::-1], R, t,
flags=cv2.CALIB_ZERO_DISPARITY,
alpha=-1)
# Compute the rigid transform that OpenCV apply to world points (USEFUL LATER)
# in order for the rectified reference camera to be K_new[I|0]
tn_1 = np.zeros((3,1)) # Cameras are never translated in the rectification
G1_rect = np.block([[R1, tn_1], [np.zeros((1,3)), 1.0]])
maps1 = cv2.initUndistortRectifyMap(K1, dcoeffs1, R1, Pn1, (1920,1080), cv2.CV_32FC1)
maps2 = cv2.initUndistortRectifyMap(K2, dcoeffs2, R2, Pn2, (1920,1080), cv2.CV_32FC1)
img1_remap = cv2.remap(img1, maps1[0], maps1[1], cv2.INTER_LANCZOS4)
img2_remap = cv2.remap(img2, maps2[0], maps2[1], cv2.INTER_LANCZOS4)
Result of the rectification:
![Rectified reference image](https://drive.google.com/open?id=10VfgXrXFO3_lYqtO9qJXr17Dc6F1PuXU)
![The other one rectified](https://drive.google.com/open?id=13ZkeMiF5xEovGmX13LSQVaJ237hoJLX0)
#....
#Now call a function that recognize a known object in the images (target)
# Find target
target_corners, _ = dt.detectTarget(img_scene1, img_target, 0.5) # return 4 corners of the detected polygon
target_corners = target_corners[:,0,:]
# Compute mask for the target cutout:
target_mask = mp.maskPolygon(target_corners, img_scene1.shape[::-1]) # Output: mask of same dimension of the image
Target found:
![Target found](https://imgur.com/QjYV8tp) [https://imgur.com/QjYV8tp](https://imgur.com/QjYV8tp)
# Compute disparity map
# https://docs.opencv.org/3.3.1/d2/d85/classcv_1_1StereoSGBM.html
window_size = 5
min_disp = 16
max_disp = 1024
num_disp = max_disp-min_disp # Deve essere divisibile per 16!
stereo = cv2.StereoSGBM_create(minDisparity = min_disp,
numDisparities = num_disp,
blockSize = window_size,
P1 = 8*3*window_size**2,
P2 = 32*3*window_size**2,
disp12MaxDiff = 1,
uniquenessRatio = 10,
speckleWindowSize = 150,
speckleRange = 2
)
print('Calcolo SGBM della disparità...')
disp = stereo.compute(img_scene1, img_scene2).astype(np.float32) / 16.0
target_disparity = target_mask*disp
points = cv2.reprojectImageTo3D(target_disparity, Q)
# DEBUG:
cv2.namedWindow('scene1', cv2.WINDOW_NORMAL)
cv2.resizeWindow('scene1', 800,450)
cv2.imshow('scene1', img_scene1)
cv2.namedWindow('disparity', cv2.WINDOW_NORMAL)
cv2.resizeWindow('disparity', 800,450)
cv2.imshow('disparity', (disp-min_disp)/num_disp)
cv2.namedWindow('target_disparity', cv2.WINDOW_NORMAL)
cv2.resizeWindow('target_disparity', 800,450)
cv2.imshow('target_disparity', target_mask*(disp-min_disp)/num_disp)
cv2.waitKey()
cv2.destroyAllWindows()
# Obtain matrix of the target 3D points starting from disparity image obtained from reprojectImageTo3D()
mask_disp = disp > disp.min()
mask_inf = ~(np.isinf(points[:,:,0]) | np.isinf(points[:,:,1]) | np.isinf(points[:,:,2]))
mask_nan = ~(np.isnan(points[:,:,0]) | np.isnan(points[:,:,1]) | np.isnan(points[:,:,2]))
mask = mask_disp & mask_inf & mask_nan
pts3D = points[mask]
Now, i have 3d reconstructed the region of the images corresponding to the target. I noted that OpenCv, during camera rectification, apply a rigid transform to world points such that the reference original camera and the new (rectified) reference camera have the same extrinsics (R=eye(3) and t=[0,0,0]'). Infact, during rectification both cameras **must be rotated**, and I think OpenCV simply brings back the new cameras to a new reference such that the reference rectified camera has the same extrinsics of the original one. **But this implies that the reconstructed 3d points will be expressed in a world reference that is not the world reference of the original camera!**
So, applying the inverse rigid transform to the pts3D, we obtain a reconstruction in the original reference camera frame:
target3Dpts_hom = cv2.convertPointsToHomogeneous(target3Dpts)[:,0,:].T
target3Dpts_hom = G.T @ target3Dpts_hom
new_target3Dpts = cv2.convertPointsFromHomogeneous(target3Dpts_hom.T[:,np.newaxis,:])[:,0,:]
Please NOTE that if I don't perform this operation, the pt3D reprojected on the original cameras by means of their projection matrices will not correspond to the target points!
Check reconstruction via reprojection; Now, i can reproject the new_target3Dpts:
Let me introduce the projection function that I call:
def proj_dist(P, dcoeffs, M):
# proj(): Esegue la proiezione prospettica dei punti 3D M secondo la MPP P,
# sul piano immagine 2D di una camera con lente.
import numpy as np
import cv2
K, R, t,_,_,_,_ = cv2.decomposeProjectionMatrix(P)
rotv, _ = cv2.Rodrigues(R)
# Proiezione. Ritorna punti immagine (N,2)
m,_ = cv2.projectPoints(M,rotv,t[0:-1],K,dcoeffs)
m = m.squeeze()
return m
Finally, the reprojections:
#P_kin = K_kin[eye(3),0] # Originals MPPs of two cameras
#P_rpi = K_rpi[R,t]
m0 = proj.proj_dist(P_kin,dcoeffs_kin,new_points).astype('int32')
for (x, y) in m0:
x = int(x)
y= int(y)
cv2.circle(img_kin, (x, y), 2, (255, 255, 0), 4)
cv2.namedWindow('frame1', cv2.WINDOW_NORMAL)
cv2.resizeWindow('frame1', 800,450)
cv2.imshow('frame1',img_kin)
cv2.waitKey(0)
But, while the reprojected points on the original reference camera are correct, this is not true for the second one....The points are simply translated, but I can't explain why.
Results: ![First frame repj](https://imgur.com/S4lo9Wz) [link text](https://imgur.com/S4lo9Wz)
![2nd frame repj. Error!](https://imgur.com/y4igaEI) [link text](https://imgur.com/y4igaEI)
Any ideas?
Thank you.
SM
m1 = proj.proj_dist(P_rpi,dcoeffs_rpi,new_points).astype('int32')
img_rpi1 = img_rpi.copy()
for (x, y) in m1:
x = int(x)
y = int(y)
cv2.circle(img_rpi1, (x, y), 2, (255, 255, 0), 4)
cv2.namedWindow('frame2', cv2.WINDOW_NORMAL)
cv2.resizeWindow('frame2', 800,450)
cv2.imshow('frame2',img_rpi1)
cv2.waitKey(0)smazzerMon, 25 Mar 2019 18:03:04 -0500http://answers.opencv.org/question/210683/How to counteract the scalling of the x and y coordinate when getting closer to the objecthttp://answers.opencv.org/question/210744/how-to-counteract-the-scalling-of-the-x-and-y-coordinate-when-getting-closer-to-the-object/At the moment I am working on a computer vision project for a asignment. In this project I am trying to create a 3d marker tracking system with stereo vision. I am able to calibrate both camera's at the same time using a chessboard pattern. After that both camera's are stereo calibrated. when the calibration is complied, both cameras video's are tracked, points undistorted and rectified. The last step is the triangulation function in OpenCV of both the marker positions. This determent the z coordinate with good enough precision. Only when a object gets closer to the camera lens the distance between the objects changes because of the pin hole principle. only when your working with body segment lengths they should not change this will lead to invalid results.
When comparing the size change of the two markers when getting closer. it seems that the change in size correlates precisely with the z coordinate behavier(see figure 1). What happend in this graph is a stick with two markers moving closer and further away from the camera lenses(see figure2)![image description](/upfiles/15536286089547407.png)
figure 1
![image description](/upfiles/15536289577118207.png)
With this knowledge there should be a way to correct the x and y position when something gets closer or further away from the lens. I tried findholography but the results where horrendous.
My question is: is there a function in OpenCV witch can adjust the scaling problem of the x and y coordinate accordingly to the z axis changes, otherwise is there a reliable methode of changing the x an y coordinates?
or am I forgetting a key point in the process, ia m quite new to computer vision and OpenCVso this could wel be the case.R.M.van_lTue, 26 Mar 2019 14:39:44 -0500http://answers.opencv.org/question/210744/Triangulate Chessboard gives weird resultshttp://answers.opencv.org/question/208725/triangulate-chessboard-gives-weird-results/Hi all,
So I'm trying to create a projection in 3D with use of stereo camera with use of Python.
I found the intrinsic camera calibration parameters, which seem good.
Then I find the chessboard corners on both images with :
`_, C1 = cv2.findChessboardCorners(img1, (6, 9), None)`.
I undistort the found corners with:
`C1Norm = cv2.undistortPoints(C1, K1, D1)`
And use those to find the essential matrix with:
`E, mask = cv2.findEssentialMat(C1Norm, C2Norm, focal=1.00, pp=(0., 0.), method=cv2.RANSAC, prob=0.999)`.
And at last I find the rotation and translation between the cameras with:
`M, R, t, mask = cv2.recoverPose(E, C1Norm, C2Norm)`.
Now with use of that I find the projection matrix of both cameras;
`P1 = K1 * [I3 | 0]` and `P2 = K2 * [R | t]` where K1 and K2 are the intrinsic camera parameters and I is an 3x3 identity matrix.
Last step now should be to triangulate. I use `4D = triangulatePoints(P1, P2, C1Trans, C2Trans)` which gives me the 4D coordinates.
However, when I plot this, my chessboard is all crooked and wrong. Has anyone any idea where something might go wrong?
Also, I know the dimensions of the chessboard, any idea how I can turn the homogeneous coordinates of the chessboard corners to cm?
BerrentMon, 11 Feb 2019 23:29:07 -0600http://answers.opencv.org/question/208725/Triangulation from spheric modelhttp://answers.opencv.org/question/208352/triangulation-from-spheric-model/Hello
I use Mei calibration which create unified camera modelisation. I want to triangulate 3D points from two known camera position and there match in two image.
Here is a draw of the problem
![image description](/upfiles/15492255887689527.png)
I can't find any information to estimate the 3D point P from two camera view.
Can someone give me article which deal with it?xavier12358Sun, 03 Feb 2019 14:28:26 -0600http://answers.opencv.org/question/208352/Correctly interpreting the Pose (Rotation and Translation) after 'recoverPose' from Essential matrixhttp://answers.opencv.org/question/208290/correctly-interpreting-the-pose-rotation-and-translation-after-recoverpose-from-essential-matrix/ Hi,
I have been breaking my head trying to correctly interpret the results of recoverPose from Essential matrix. Here are the high level steps I am using:
1. Detect ORB features in two images
2. Match featues using BFMatcher
3. findEssential across two images
4. recoverPose ie. R,T from the two images
5. Triangulate the good featues (masked from recoverPose) using the R, T to created 3d point-clouds (landmarks)
6. As a ground truth, I also extract Chess board corners from the image and triangulate them using the R, T calculated above. A good planar formation for chess board corners indicates that R, T are accurate for triangulation.
7. Plot everything
**So as we can see from images 1488 and 1490, the camera is moving to the left - up AND it in pointing down and to the right. However the plot of R and T of the 2nd position reflects something completely different.**
![image description](/upfiles/15490517252702959.png)
![image description](/upfiles/15490517382481242.png)
I have tried inverting both using R' and -(R')*T, but that doesn't plot correctly either. I have tried a bunch of different combinations, but none seem to make sense.
So what gives???
The python script and test images can be found [here](https://drive.google.com/drive/folders/1wrYzphphyrFkUIiqXpfm-2mBQsrAN03b?usp=sharing). For reference the python code is:
import numpy as np
import cv2
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
def plot_pose3_on_axes(axes, gRp, origin, axis_length=0.1):
"""Plot a 3D pose on given axis 'axes' with given 'axis_length'."""
# get rotation and translation (center)
#gRp = pose.rotation().matrix() # rotation from pose to global
#t = pose.translation()
#origin = np.array([t.x(), t.y(), t.z()])
# draw the camera axes
x_axis = origin + gRp[:, 0] * axis_length
line = np.append(origin, x_axis, axis=0)
axes.plot(line[:, 0], line[:, 1], line[:, 2], 'r-')
y_axis = origin + gRp[:, 1] * axis_length
line = np.append(origin, y_axis, axis=0)
axes.plot(line[:, 0], line[:, 1], line[:, 2], 'g-')
z_axis = origin + gRp[:, 2] * axis_length
line = np.append(origin, z_axis, axis=0)
axes.plot(line[:, 0], line[:, 1], line[:, 2], 'b-')
img1 = cv2.imread('/home/vik748/data/chess_board/GOPR1488.JPG',1) # queryImage
img2 = cv2.imread('/home/vik748/data/chess_board/GOPR1490.JPG',1)
fx = 3551.342810
fy = 3522.689669
cx = 2033.513326
cy = 1455.489194
K = np.float64([[fx, 0, cx],
[0, fy, cy],
[0, 0, 1]])
D = np.float64([-0.276796, 0.113400, -0.000349, -0.000469]);
print(K,D)
# Convert images to greyscale
gr1=cv2.cvtColor(img1,cv2.COLOR_BGR2GRAY)
gr2=cv2.cvtColor(img2,cv2.COLOR_BGR2GRAY)
#Initiate ORB detector
detector = cv2.ORB_create(nfeatures=25000, edgeThreshold=15, patchSize=125, nlevels=32,
fastThreshold=20, scaleFactor=1.2, WTA_K=2,
scoreType=cv2.ORB_HARRIS_SCORE, firstLevel=0)
# find the keypoints and descriptors with ORB
kp1, des1 = detector.detectAndCompute(gr1,None)
kp2, des2 = detector.detectAndCompute(gr2,None)
print ("Points detected: ",len(kp1), " and ", len(kp2))
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1,des2)
kp1_match = np.array([kp1[mat.queryIdx].pt for mat in matches])
kp2_match = np.array([kp2[mat.trainIdx].pt for mat in matches])
kp1_match_ud = cv2.undistortPoints(np.expand_dims(kp1_match,axis=1),K,D)
kp2_match_ud = cv2.undistortPoints(np.expand_dims(kp2_match,axis=1),K,D)
E, mask_e = cv2.findEssentialMat(kp1_match_ud, kp2_match_ud, focal=1.0, pp=(0., 0.),
method=cv2.RANSAC, prob=0.999, threshold=0.001)
print ("Essential matrix: used ",np.sum(mask_e) ," of total ",len(matches),"matches")
points, R, t, mask_RP = cv2.recoverPose(E, kp1_match_ud, kp2_match_ud, mask=mask_e)
print("points:",points,"\trecover pose mask:",np.sum(mask_RP!=0))
print("R:",R,"t:",t.T)
bool_mask = mask_RP.astype(bool)
img_valid = cv2.drawMatches(gr1,kp1,gr2,kp2,matches, None,
matchColor=(0, 255, 0),
matchesMask=bool_mask.ravel().tolist(), flags=2)
plt.imshow(img_valid)
plt.show()
ret1, corners1 = cv2.findChessboardCorners(gr1, (16,9),None)
ret2, corners2 = cv2.findChessboardCorners(gr2, (16,9),None)
corners1_ud = cv2.undistortPoints(corners1,K,D)
corners2_ud = cv2.undistortPoints(corners2,K,D)
#Create 3 x 4 Homogenous Transform
Pose_1 = np.hstack((np.eye(3, 3), np.zeros((3, 1))))
print ("Pose_1: ", Pose_1)
Pose_2 = np.hstack((R, t))
print ("Pose_2: ", Pose_2)
# Points Given in N,1,2 array
landmarks_hom = cv2.triangulatePoints(Pose_1, Pose_2,
kp1_match_ud[mask_RP[:,0]==1],
kp2_match_ud[mask_RP[:,0]==1]).T
landmarks_hom_norm = landmarks_hom / landmarks_hom[:,-1][:,None]
landmarks = landmarks_hom_norm[:, :3]
corners_hom = cv2.triangulatePoints(Pose_1, Pose_2, corners1_ud, corners2_ud).T
corners_hom_norm = corners_hom / corners_hom[:,-1][:,None]
corners_12 = corners_hom_norm[:, :3]
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.set_aspect('equal') # important!
title = ax.set_title('3D Test')
ax.set_zlim3d(-5,10)
# Plot triangulated featues in Red
graph, = ax.plot(landmarks[:,0], landmarks[:,1], landmarks[:,2], linestyle="", marker="o",color='r')
# Plot triangulated chess board in Green
graph, = ax.plot(corners_12[:,0], corners_12[:,1], corners_12[:,2], linestyle="", marker=".",color='g')
# Plot pose 1
plot_pose3_on_axes(ax,np.eye(3),np.zeros(3)[np.newaxis], axis_length=0.5)
#Plot pose 2
plot_pose3_on_axes(ax, R, t.T, axis_length=1.0)
ax.set_zlim3d(-2,5)
ax.view_init(-70, -90)
plt.show()vik748Fri, 01 Feb 2019 14:22:32 -0600http://answers.opencv.org/question/208290/