OmniTry does mask-free virtual try-on by finding the spot itself

If you have ever tried to "try on" glasses or a tie on your own photo in an app, you know the catch: the system needs you to point out the region to replace by hand — draw a mask or a box. Across hundreds of item types that is awkward and scales badly. OmniTry takes a different route: the model finds the place where the object logically belongs and puts it there, with no masks and no extra hints. And it works not only on clothing, but on jewelry, bags, hats, glasses and even unusual objects.

Why this was hard

Classic VTON systems lean on large paired datasets: a photo of a person "before" and a correct try-on "after". For clothing such sets are at least collectable; for earrings, belts or ties in real poses they barely exist. And the user is not a retoucher — asking them to draw masks is a dead end. OmniTry is built as a universal "anything try-on": the model sees a photo of a person and a photo of the product, decides for itself where and how to place the object, and leaves everything else in the shot untouched.

How it works under the hood

The core idea is two-stage training on a diffusion transformer (DiT). In the first stage the model learns a sense of localization: where and at what scale an item logically goes. For that the authors take large sets of unpaired in-the-wild portraits and automatically build training triplets — the original, a version with the object erased, and a short description of the item. They repurpose an inpainting model here: it is fed a zero mask, so in effect it learns to add an object without knowing its contours in advance. Local edits come out natural, and the rest of the face and the scene stays untouched.

Keeping the model from cheating

Naive erasure leaves invisible traces that let the network guess the position and the shape — and stop generalizing. OmniTry introduces trace-free erasure: after inpainting, the image is lightly repainted in image-to-image mode and softly blended with the original outside the region. The traces disappear, and the model has to actually learn localization from context rather than from accidental cues.

How the model remembers a specific product

The second stage is about item identity: the glasses on the person should be those exact glasses from the storefront. Here the authors add a small amount of paired data and a new attention scheme. The product image tokens enter the transformer alongside the person tokens, but with a guard against unwanted information flow: the path from the try-on result back to the object is blocked so the product's details are not washed out. For flexibility there are two LoRA adapters — a location one (from the first stage) and an identity one (for transferring the object's appearance). Together they make fine-tuning fast even with a few dozen pairs for a rare class.

Where it was tested

The team assembled OmniTry-Bench, a single benchmark covering 12 object types: from tops and dresses to shoes, earrings, rings, bags, belts, glasses, hats and ties. It includes shots on white and natural backgrounds, product photos and even already-tried-on images as input — 360 carefully selected pairs in all. Evaluation runs on three groups of metrics: object consistency (DINO and CLIP), person preservation (LPIPS, SSIM) and correct localization (a detector plus a text check of descriptions from a multimodal LLM).

What it does in practice

On the full set OmniTry reaches M-DINO 0.6160 and M-CLIP-I 0.8327 with very good person preservation (LPIPS 0.0542, SSIM 0.9333) and near-flawless localization (G-Acc. 0.9972). That beats both mask-free generators and many methods that were handed hand-drawn masks. On the clothing subset the numbers are higher still (M-DINO 0.6995, for instance), and the system holds up on in-the-wild backgrounds and varied input formats. One important finding is the few-shot effect: thanks to the first stage, literally a handful of pairs for a rare class is enough to bring identity fidelity up quickly.

Why it matters

• For online retail: less manual annotation and faster launches of new categories.
• For creators and brands: natural try-ons in real poses, with no studio sessions.
• For AR/VR: one framework for clothing, jewelry and accessories, with no dependence on human parsers.
• For robustness on rare classes: trace-free erasure and two-stage training help generalization.

Limits and what comes next

OmniTry can still go wrong under heavy occlusion (complex hand poses with jewelry, for example), with very small objects and at extreme viewing angles. The long tail of rare products needs active fine-tuning, and contact physics — wrinkled fabric, shadows cast by earrings — calls for more explicit 3D cues. Next steps worth taking: better multimodal prompting, accounting for materials and lighting, and evaluating fairness and bias across different appearance types. Code, weights and the benchmark are available: https://omnitry.github.io/.

How this connects to current AI trends

OmniTry builds on diffusion transformers with rotary positional embeddings (RoPE) and rectified flow, combines inpainting with full attention to the object, and makes careful use of multimodal prompts from an MLLM (multimodal large language model) for descriptions. That unified approach fits well with multi-agent systems, where one agent prepares the data and the text and another assembles the final try-on. It is an example of how careful data and architecture engineering can replace clunky masks and hand-drawn contours — and make virtual try-on genuinely universal.