How garment segmentation actually works

Segmentation is the step where a system decides which pixels in an image belong to the garment and which do not. Everything else in a clothing pipeline tends to sit downstream of it: measuring a length, transferring a garment onto another body, cataloguing a shape, or comparing two items all begin with a mask. It is worth understanding what a mask is good at and where it reliably breaks, because the failures are not evenly distributed across the garment.

What the model is being asked

A segmentation model assigns a label to every pixel. In the simplest version there are two labels, garment and not-garment. In practice most clothing work uses more: upper body, lower body, outerwear, footwear, skin, hair, background. The model learns these from images that human annotators have traced by hand, and the quality of that tracing sets the ceiling on everything the model can do.

This matters more than it sounds. Annotators tracing a jacket over a jumper have to decide, thousands of times, where one garment stops and the other begins in a two-dimensional image where they overlap. Different annotators decide differently at the cuff, at the collar and wherever a layer is partly hidden. The model learns the average of those decisions, which is to say it learns to be confidently vague in exactly the places where layering happens.

The easy boundary and the hard one

A sleeve edge against a plain background is close to trivial. There is a sharp change in colour and texture across a smooth curve, and every method from the last two decades handles it. The difficulties are elsewhere and they have a pattern.

• Hem over trousers. A shirt hem falling over a waistband produces two fabrics of similar tone meeting at a soft, folded, often shadowed line. There is frequently no visual edge to find at all.
• Hair over shoulders. Long hair crosses the garment boundary in thin strands with partial transparency. Every strand is a decision, and a mask that is a few pixels wrong here looks obviously wrong to a person.
• Self-shadow in folds. A deep fold reads as a dark region that a model may assign to background, punching holes in the middle of an otherwise clean mask.
• Sheer and open-weave cloth. Fabric you can partly see through is genuinely ambiguous: the honest answer is a fractional value per pixel, and a hard label cannot express it.
• Prints that resemble the background. A patterned garment photographed against a busy scene loses the texture cue the model relies on most.

Why the errors are worse than they look

A mask that is ninety-something per cent correct by area can still be unusable, because the wrong pixels are concentrated at the boundary and the boundary is what everything downstream uses. A length measured from a mask is measured between two boundary points. A garment transferred onto another body is composited along its boundary. Interior pixels, which dominate any area-based score, contribute almost nothing to either task.

Interior pixels dominate the score and contribute almost nothing to the task.

This is why boundary-focused evaluation exists, and why a system that reports only an overlap score is telling you the least informative version of its own performance. If a mask is going to be used for measurement, it should be evaluated on the boundary; if it is going to be used for compositing, it should be evaluated on how it handles transparency and thin structures.

The practical response

Most of the improvement available here is not model architecture. It is controlling the image. A garment photographed flat against a contrasting plain surface, evenly lit so folds do not read as background, and shot square to the camera, will segment far better than the same garment worn in a room. When the photograph cannot be controlled, the useful move is to keep the uncertainty rather than discarding it: a soft mask that says a pixel is partly garment carries information a hard mask throws away, and the next stage can decide what to do with it.

The general shape of the lesson recurs throughout this field. The interesting work is in the ten per cent of the image where the answer is genuinely ambiguous, and the standard metric is dominated by the ninety per cent where it never was.