Building an adjacency matrix from Harris CD output

edit

My final intention is to build something based off this : https://www.researchgate.net/publicat...

And I'm stuck at part 3.2, with building the adjacency matrix for the corners. The figure below is to help vizualize what I'm trying to achieve.

Say I take that image of A and apply Harris to it and then use this code

# define the criteria to stop and refine the corners
ret, labels, stats, centroids = cv2.connectedComponentsWithStats(dst)
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.001)
corners = cv2.cornerSubPix(gray,np.float32(centroids),(5,5),(-1,-1),criteria)

#print corners
for i in range(1, len(corners)):
    print(corners[i])

I get a nice list of corner coordinates that looks like this :

[51.70207 62.5541 ]
[ 37.04742 100.78287]
[ 66.707726 100.73132 ]
[ 29.685337 119.28638 ]
[ 75.62286 119.49719]

For reference, the image is 106 x 189. But how can I figure out which is connected to which? I tried "walking" the image array from a corner until I run into another but this doesn't cover cases where there are multiple. I also thought about how I can iterate over each corner and see if I can connect to others, but since they're part of the same line they will always be able to connect.

Any advice is appreciated.