# How to merge nearby rectangles

I am working on character segmentation and looking to merge nearby characters into a box. I have successfully found the contours but I am not sure how to find a straight line between the top and bottom points.

def findContours(self, image):
contour_img = image.copy()
vis = image.copy()
vis = cv2.cvtColor(vis, cv2.COLOR_GRAY2BGR)
_, contours, hierarchy = cv2.findContours(contour_img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

contoursBBS = []
for contour in contours:
[x, y, w, h] = cv2.boundingRect(contour)

if w > 100 or h < 8:
continue

contoursBBS.append([x, y, w, h, w*h])
contoursBBS = np.array(contoursBBS)

# reject outliers based on width, height, and area
rejectH = self.rejectOutliers(contoursBBS, 3, m=4)
rejectW = self.rejectOutliers(rejectH, 2, m=4)
rejectA = self.rejectOutliers(rejectW, 4, m=4)

contourRects = []
for c in rejectA:
[x, y, w, h, a] = c
if w < 9 or h < 15 or a < 300 or a > 6000:
continue
contourRects.append(c)

for i, rect in enumerate(contourRects):
[x, y, w, h, a] = rect
topCenter, bottomCenter = (int((2*x + w)/2), int(y)), (int((2*x + w)/2), int(y+h))

print("X: {:4d}  Y:  {:4d}  W: {:4d}  H: {:4d}  A: {:4d}".format(x, y, w, h, a))

cv2.rectangle(vis, (x, y), (x+w, y+h), (0, 255, 0), 2)
cv2.circle(vis, topCenter, 2, (0, 0, 255), 2)
cv2.circle(vis, bottomCenter, 2, (0, 0, 255), 2)


Result

What I am hoping to achieve

Some test images (requested by @supra56)

edit retag close merge delete

Sort by » oldest newest most voted

untested, but you could add all the contour points to a single array, and call minAreaRect on it:

arr = []
for x,y,w,h in contourRects:
arr.append((x,y))
arr.append((x+w,y+h))

box = cv.minAreaRect(np.asarray(arr))
pts = cv.boxPoints(box) # 4 outer corners

more

I tried but it computes the average rectangles, like this (the blue rectangle) https://ibb.co/H2JjD1B

( 2018-11-26 23:42:50 -0500 )edit

you seem to do that with your "center" points, not with the actual contour ones. you also have to get rid of the 2 outlier rects at the left / right side first

( 2018-11-26 23:45:57 -0500 )edit

I tried again with actual contours ones and the results are better. Can you show me how to get rid of the outliers?

( 2018-11-26 23:51:35 -0500 )edit

maybe filter for large height ?

( 2018-11-27 00:00:12 -0500 )edit

characters have similar size, how about pick median value and filter by a multiplemedian e.g. (height > 0.75median && height < 1.25*median)

( 2018-11-27 00:25:00 -0500 )edit

@berak My application will process real time images of different heights, so I’ll have to figure out a dynamic algorithm for an optimal height. @blues I’ll try this later. Thanks!

( 2018-11-27 00:27:50 -0500 )edit

btw, we do have proper text detection methods builtin.

( 2018-11-27 00:36:40 -0500 )edit

Thanks for the information. But I found that EAST will introduce extra computational costs.

( 2018-11-27 00:41:31 -0500 )edit

@Rex Low. Can you post orginal image?

( 2018-11-29 05:41:51 -0500 )edit
1

@supra56 Hi, sorry for late response. I have updated the post with several test images. Please have a look at them :)

( 2018-12-02 05:59:33 -0500 )edit

Append all your rectangles in rectList and use cv2.boundingRect() to get the bigger rectangle that enclose all sub rectangles. Let me know if it works for you. Thanks berak's for the piece of code


arr = []
for x,y,w,h in rectList:
arr.append((x,y))
arr.append((x+w,y+h))

x,y,w,h = cv2.boundingRect(np.asarray(arr))
cv2.rectangle(img,(x,y),(x+w,y+h),(0,0,255),1)

more

I found better solution without iterating through contours - by using numpy concatenate() function:

concat = np.concatenate(contours)
hulls = cv2.convexHull(concat)


Resulting hulls contour can be further approximated by cv2.approxPolyDP() function in order to get bounding rectangle.

more

Official site

GitHub

Wiki

Documentation