Why findFundamentalMat gives different results for same but different orientation of points?

edit

Sorry if the title is kind of weird. It is quite hard to express the question of my problem.

So, I am in the middle of a 3D reconstruction project. The pipeline is more or less the same with the standard pipeline where

1. Undistort image
2. Detect points with keypoint detector
3. Track the points across frames (optical flow)
4. Calculate the fundamental matrix

and so on. The only different part is at step 2 where I use a Line Segment Detector and track it across frames.

So, if I am using a keypoint detector, giving two frame of images, I will get two set of keypoints (each set corresponds to each frame). But as for my situation, I have four set of keypoints (each two set correspond to each frame since a line has a start point and an end point).

In order to calculate the Fundamental matrix, I need to concatenate the two sets of point of each frame.

One way is by just vertically concatenate it: np.vstack([start_point, end_point]).

The other way is by :np.hstack([start_point, end_point]).reshape(-1, 2). Means, it is concatenated 'alternately', i.e.

[[start_point[0],
  end_point[0],
  start_point[1],
  end_point[1],
             ...]]

Both will end up with a same shape. But fair enough, they produce a quite different results. From my observation, the vstack produced a more '3D-like' result while the hstack produced a more 'planar-like' result for the reconstruction.

The question is why is this? And which one supposed to be better?

Below is sample code to give a view of this question:

import numpy as np
import cv2

np.random.seed(0)

def prepare_points(pts_frame1, pts_frame2):
    # Prepare the four sets of points
    (p1_f1, p2_f1) = pts_frame1
    (p1_f2, p2_f2) = pts_frame2

    v_stacked_f1f2 = (np.vstack([p1_f1, p2_f1]), np.vstack([p1_f2, p2_f2]))
    h_stacked_f1f2 = (np.hstack([p1_f1, p2_f1]).reshape(-1, 2), 
                                np.hstack([p1_f2, p2_f2]).reshape(-1, 2))

    return (v_stacked_f1f2, h_stacked_f1f2)

pts_frame1 = np.random.random_sample((60, 2)).astype("float32")
pts_frame2 = np.random.random_sample((60, 2)).astype("float32")

# Emulate the two sets of points for each frame where
# the first set is the start point, while
# the second set is the end point of a line
pts_frame1 = (pts_frame1[::2], pts_frame1[1::2])
pts_frame2 = (pts_frame2[::2], pts_frame2[1::2])

(v_stacked_f1f2, h_stacked_f1f2) = prepare_points(pts_frame1, pts_frame2)

F_vstacked = cv2.findFundamentalMat(v_stacked_f1f2[0], v_stacked_f1f2[1],
                                    cv2.FM_RANSAC, 3, 0.99)[0]
F_hstacked = cv2.findFundamentalMat(h_stacked_f1f2[0], h_stacked_f1f2[1],
                                    cv2.FM_RANSAC, 3, 0.99)[0]

print("F_vstacked:\n", F_vstacked, "\n")
print("F_hstacked:\n", F_hstacked, "\n")

# The output:    
# F_vstacked:
# [[ 3.31788127 -2.24336615 -0.77866782]
# [ 0.83418839 -1.4066019  -0.92088302]
# [-2.75413748  2.27311637  1.        ]] 

# F_hstacked:
# [[  7.70558741  25.29966782 -16.20835082]
# [-12.95357284  -0.54474384  14.95490469]
# [  1.79050172 -10.40077071   1.        ]]