# Is there an ellipse detection algorithm that is robust to partially occluded objects?

I would like to use OpenCV to track an animal's pupil in a video recording of a mouse viewing naturalistic visual scenes. Here is an example of what a given image might look like under the best of circumstances and another example of poor quality.

My impression from browsing the Q&A is that ellipse detection is the problem I will need to solve for this task. So far I have tried this process:

1. Process the image into a binary format

import cv2 as cv
gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY) # convert to grayscale
blur = cv. medianFilter(gray, 11) # blur out some of the edges
(threshold, binary) = cv.threshold(blur, 50, 255, cv.THRESH_BINARY_INV) # binarize

2. Extract contours from the binary image

(image2, contours, hierarchy) = cv.findContours(binary, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_NONE)

3. Identify the contour most likely to represent the pupil

import numpy as np
(xc, yc) = np.array(gray.shape) / 2
dxy = [np.sqrt((yc - cnt.squeeze().mean(0)[1]) ** 2 + (xc - cnt.squeeze().mean(0)[0]) ** 2 for cnt in contours]
best = np.argsort(dxy)[0] # this is the contour closest to the center of the image
pupil = contours[best]

4. Fit the best contour with an ellipse

(xc, yc), (a, b), theta = cv.fitEllipse(pupil)


This approach works well for images in which the pupil is entirely visible. Here is an example of the processing and result of this procedure.

Like I said this works pretty well for cases in which the pupil is entirely visible, but for cases in which the pupil is partially occluded by the animal's eyelid, the algorthim breaks down.

I've also tried using a brute force approach (i.e., Hough circle transform), but it seems like this approach depends on the assumption that the object is perfectly circular which the pupil is not because of the angle of the camera. This approach is also computationally expensive which makes it suboptimal for processing long video recordings acquired at a high-frequency sampling rate (1 hour long recordings at 120 - 200 fps in my case).

I came across some more recently developed procedures for ellipse detection which are specifically robust to the case of partially occluded shapes. This paper is from 2006 and this paper is from 2014. These algorithms seem particuarly effective for detection of partially occluded ellipses; however, they don't have an implementation in OpenCV.

My question is two (and a half) parts. Is it possible to create an algorithm that makes detection of ellipses robust to partial occlusion using only the existing tools availbale in OpenCV, and if so, how would you approach this aim? Second, would it be feasible to implement the geometric algorithms for ellipse detection (referenced in the papers I linked) in OpenCV? To this second point, I would love to contribute this additional functionality to OpenCV, but I have never programmed in c++; however, I have had some experience contributing to existing projects written in Python. Is it possible to contribute ...

edit retag close merge delete

improve the lighting. not more light, different light. that reflection on the surface of the eye has to go. move the light so that reflection isn't near the pupil. then, find a threshold that only selects the very dark pupil, not any somewhat dark areas. experiment with a handheld flashlight. once you have the pupil contour, you can deal with occlusion. let's assume most of it is still oval, and maybe some of it is clipped and somewhat straight (eye lid). you could try detecting low curvature points and remove them (and consider it a set of points rather than a sequence of lines), or you could fit a minimal enclosing rectangle/circle and try to work from that.

( 2020-10-11 15:05:00 -0500 )edit

I just noticed there's fitEllipsethat can probably help you, but only if your ellipse is clipped, not when other stuff is stuck to it https://docs.opencv.org/3.4/de/d62/tu...

( 2020-10-12 16:23:56 -0500 )edit

To your first point, my intuition is that the reflection of the light source isn't the primary barrier to accurate and precise ellipse detection. It may sometimes occlude a portion of the pupil's edge (e.g., when the animal looks forward or nasally), but I think the percent of the pupil's edge lost to the reflection of the light source is marginal. I agree that changing the threshold might help distinguish the pupil from the eyelid. I can play around with the threshold, but I think ideally I would have a formula for determining the optimal threshold for a given frame. How might you formulate the thresholding given the statistics of a single image? To your second point, can you explain what you mean by "clipped" and how what you are suggesting is different than step 4 in my procedure?

( 2020-10-13 15:15:45 -0500 )edit

you can try otsu thresholding, or just fix the intensity of lighting and camera's aperture/exposure time/gain so nothing changes on its own, and then "empirically" choose a threshold that works for you. there are more complicated things to try, but those are to try and ad hoc. as for fitellipse, I only noticed later that you're already using that function. clipped meaning a segment from the ellipse is missing/occluded. disregard if that doesn't clarify. it's not that important. THE most important part is lighting. least effort, most effect. after the picture is taken, anything to be corrected takes orders of magnitude more effort. that includes lighting such that only the pupil is dark and eyelids aren't.

( 2020-10-15 06:09:28 -0500 )edit

Okay, this helps a lot. Thank you. I'm going to play around with the lighting. If I can find an optimal position for the light source, I'll update my post with the solution. Thanks again.

( 2020-10-15 11:46:25 -0500 )edit

also try modifying the light source. a "softbox" is a common method to make glare less noticeable, and also illuminate the object more evenly.

( 2020-10-16 11:17:11 -0500 )edit