Sorry if the title is kind of weird. It is quite hard to express the question of my problem.
So, I am in the middle of a 3D reconstruction project. The pipeline is more or less the same with the standard pipeline where
- Undistort image
- Detect points with keypoint detector
- Track the points across frames (optical flow)
- Calculate the fundamental matrix
and so on. The only different part is at step 2 where I use a Line Segment Detector and track it across frames.
So, if I am using a keypoint detector, giving two frame of images, I will get two set of keypoints (each set corresponds to each frame). But as for my situation, I have four set of keypoints (each two set correspond to each frame since a line has a start point and an end point).
In order to calculate the Fundamental matrix, I need to concatenate the two sets of point of each frame.
One way is by just vertically concatenate it: np.vstack([start_point, end_point]).
The other way is by :np.hstack([start_point, end_point]).reshape(-1, 2). Means, it is concatenated 'alternately', i.e.
[[start_point[0],
end_point[0],
start_point[1],
end_point[1],
...]]
Both will end up with a same shape. But fair enough, they produce a quite different results. From my observation, the vstack produced a more '3D-like' result while the hstack produced a more 'planar-like' result for the reconstruction.
The question is why is this? And which one supposed to be better?
Below is sample code to give a view of this question:
import numpy as np
import cv2
np.random.seed(0)
def prepare_points(pts_frame1, pts_frame2):
# Prepare the four sets of points
(p1_f1, p2_f1) = pts_frame1
(p1_f2, p2_f2) = pts_frame2
v_stacked_f1f2 = (np.vstack([p1_f1, p2_f1]), np.vstack([p1_f2, p2_f2]))
h_stacked_f1f2 = (np.hstack([p1_f1, p2_f1]).reshape(-1, 2),
np.vstack([p1_f2, p2_f2]).reshape(-1, 2))
return (v_stacked_f1f2, h_stacked_f1f2)
pts_frame1 = np.random.random_sample((60, 2)).astype("float32")
pts_frame2 = np.random.random_sample((60, 2)).astype("float32")
# Emulate the two sets of points for each frame where
# the first set is the start point, while
# the second set is the end point of a line
pts_frame1 = (pts_frame1[::2], pts_frame1[1::2])
pts_frame2 = (pts_frame2[::2], pts_frame2[1::2])
(v_stacked_f1f2, h_stacked_f1f2) = prepare_points(pts_frame1, pts_frame2)
F_vstacked = cv2.findFundamentalMat(v_stacked_f1f2[0], v_stacked_f1f2[1],
cv2.FM_RANSAC, 3, 0.99)[0]
F_hstacked = cv2.findFundamentalMat(h_stacked_f1f2[0], h_stacked_f1f2[1],
cv2.FM_RANSAC, 3, 0.99)[0]
print("F_vstacked:\n", F_vstacked, "\n")
print("F_hstacked:\n", F_hstacked, "\n")
# The output:
# F_vstacked:
# [[ 3.31788127 -2.24336615 -0.77866782]
# [ 0.83418839 -1.4066019 -0.92088302]
# [-2.75413748 2.27311637 1. ]]
# F_hstacked:
# [[ 2.5356868 -0.88369618 -0.82700244]
# [ 0.00495862 1.41509886 -1.04054715]
# [-1.34766744 -0.29465906 1. ]]