This forum is disabled, please visit https://forum.opencv.org
 | 1 | initial version |
First, no rectangles are concave.
A quadrilateral is concave if and only if it has one interior angle over 180 degrees.
How smart and fast do you need?
Off the top of my head, you could find vectors for all 4 sides (not corners)* and take the outer product of each pair of adjacent sides (Outer Product is something like f([x1,y1],[x2,y2]) = x1y2+x2y1,* watch your order and direction, use the right hand rule). If the product is negative for one adjacent pair, that angle is over 180 degrees, you've got a concave quad. (if all the outer products are positive, the quad is convex) I just learned this technique myself a few days ago, so please Google for more detail first.
** subtract adjacent corners, in order CCW around the quad to get the side vectors
** I think this formula is actually the 3-D cross product in the plane z=0, by checking the sign you're seeing which direction the 3-D normal is pointing
