What a virtual try-on model is really doing

Virtual try-on produces an image: this garment, on this person, rendered plausibly enough to be worth looking at. The results have improved a great deal, to the point where the seams, the folds and the fall of the cloth can look entirely convincing. It is worth being clear about what that convincingness does and does not establish.

The pipeline in outline

Most approaches share a structure. First the person image is parsed: the body is segmented, a pose is estimated, and the region the garment will occupy is identified and removed. Then the garment image is warped, deformed from its flat or hanging state into the shape it would take on that body in that posture. Finally the warped garment is composited into the cleared region and the result is refined so that lighting, shadow and edges agree.

Each stage has its own characteristic failure. Parsing fails on layering and on hair. Warping fails on garments with strong internal structure, because a printed logo or a check pattern must deform consistently and a warp that looks smooth overall can shear a straight line. Compositing fails at boundaries, particularly where the garment meets skin or hair, and where the removed region was larger than the new garment covers.

Plausible is not fitted

The important point is what the warping stage optimises for. It is trained to produce an image that looks right: the garment follows the body's contours, the folds fall where folds fall, the result is judged against photographs of people wearing clothes. It is not trained to refuse. If the garment is two sizes too small, the warp will still stretch it onto the body and the result will still look like a garment being worn, because every training example was a garment being worn.

The warp is trained to produce a plausible image. It is not trained to refuse.

A too-tight garment in reality does something a warp does not reproduce unless it is explicitly modelled: it strains, it pulls at the fastenings, it rides up, it produces drag lines from the points of tension, and it changes the body's outline. A too-large garment collapses into folds that depend on the cloth's weight and stiffness. These are physical consequences of a garment that does not fit, and they are precisely the visual evidence a person would use to judge fit from a photograph. An image model asked to produce a pleasing composite has every incentive to smooth them away.

What the image is good for

This is not a dismissal. Try-on imagery answers real questions. Does this colour work against this skin tone and this hair. Does this length sit well on this body. Does the proportion of this jacket suit this frame. Does this pattern read as busy at this scale. These are questions of appearance, and appearance is exactly what an image model is competent at. For a person deciding between two colourways or two lengths, a good try-on image is genuinely informative.

The question it does not answer is whether the garment will fit, and the danger is that a photorealistic image implies an answer it never computed. The more convincing the rendering, the stronger the implication.

Keeping the two apart

The cleaner design keeps appearance and fit as separate outputs from separate reasoning. Fit is a comparison of measurements, garment dimensions against body dimensions, with the ease the garment was drafted with taken into account, and it produces a statement about numbers. Appearance is an image. A system that shows both, and is explicit that the image is showing how a garment looks rather than proving how it fits, is telling the truth about its own machinery.

A system that shows only the image is inviting the viewer to read a measurement out of a picture, which is the thing people are least reliable at doing and the reason the tape measure was invented.