Edge Function

For an edge running from point A to point B, and a test point P, the edge function is defined as the 2D cross product of (B - A) and (P - A): edge(A, B, P) = (B.x - A.x) * (P.y - A.y) - (B.y - A.y) * (P.x - A.x) Geometrically, this value is: * > 0 if P is to the right of the directed line A->B * = 0 if P is exactly on the line A->B * < 0 if P is to the left of the directed line A->B

If we compute this for all three edges of a triangle (A->B, B->C, C->A) and the point P gives the SAME sign for all three, then P must be inside the triangle (it's consistently on the same side of every edge, which is only possible from inside a convex shape like a triangle).

This also conveniently gives us "barycentric weights / coordinates" for free - the three edge function results, once normalised by the triangle's total area, tell us how much each vertex (A, B, C) contributes to point P. We use this to blend/interpolate the three vertex colours smoothly across the triangle.

static float edge_function(Vec2 a, Vec2 b, Vec2 p)
{
    return (b.x - a.x) * (p.y - a.y) - (b.y - a.y) * (p.x - a.x);
}