How Fine-Tuned Diffusion Models and Edge Deployment Made Our Client's Virtual Try-On App Actually Real-Time

Background

Our client's app is an AI fashion assistant: upload a photo, pick an outfit, and get a photorealistic try-on in seconds. It's built for two audiences — consumers deciding whether something's worth buying before they check out, and e-commerce brands who want on-model product photos without a photoshoot.

Both use cases live and die on the same thing: does the try-on look real, and does it come back fast enough that people don't bounce.

The challenge

Our client came to us with a working diffusion-based try-on pipeline that needed to go from "implemented" to "production-ready and fast." Three problems stood out:

1. Generation was too slow for a real shopping flow. A diffusion model that takes too long to render an outfit stops feeling like a shopping tool and starts feeling like a chore — every extra second is a chance for someone to close the tab.
2. The pipeline was doing image generation, and only image generation. Turning "a photo and a link to a pair of jeans" into a finished try-on meant tying the model together with product search/retrieval and language understanding, not just running inference.
3. Nothing was optimized for how it would actually be served. A model that works in a notebook isn't the same as a model that can serve thousands of concurrent users at low latency without burning through GPU budget.

Our approach

We treated this as three connected problems rather than one big model swap. First, we fine-tuned CatVTON — the diffusion model handling the actual garment-fitting step — using LoRA and DoRA (Weight-Decomposed Low-Rank Adaptation) adapters, so we could specialize it for realistic fit and drape without the cost of retraining the full model from scratch. Alongside that, we fine-tuned the LLM layer responsible for understanding outfit requests and generating recommendations, and integrated search APIs so the product could go from a pasted link or photo straight to a garment the model could actually use.

Once the pipeline itself was solid, we moved inference out to the edge on RunPod, so generation happens close to the user instead of round-tripping to a single central server.

How we built it

Fine-tuning CatVTON with LoRA and DoRA

CatVTON handles the core try-on generation — taking a person's photo and a garment and producing a realistic result. Rather than fully retraining it, we used LoRA and its newer variant DoRA to fine-tune it efficiently: DoRA decomposes each weight into magnitude and direction and only fine-tunes those components, which gets closer to full fine-tuning quality at a fraction of the training cost. That let us specialize the model for fit, drape, and realism without the time or GPU spend of training from scratch — and kept the adapters small enough to swap or update quickly as the product evolved.

Fine-tuning the LLM layer

On top of the visual model, we fine-tuned an LLM to handle the language side of the product — interpreting what a user is asking for and generating outfit recommendations and styling feedback, rather than treating try-on as a one-shot image-in, image-out task.

Wiring in search APIs

To let someone go from "a link to a product page" to "a try-on result," we integrated search APIs into the pipeline so the right garment image could be found and normalized automatically, instead of requiring a clean, pre-cropped product photo every time.

Deploying at the edge on RunPod

With the model and pipeline itself optimized, we deployed inference on RunPod's edge infrastructure. That meant GPU capacity could scale up and down with demand rather than sitting on always-on servers, and it put inference closer to users — which mattered a lot once we were chasing seconds, not minutes.

Results

• Try-ons that actually feel instant. End-to-end, a full outfit application now takes 4-7 seconds — fast enough to hold up in a real mobile shopping flow instead of feeling like a loading screen.
• One connected pipeline instead of three disconnected pieces. The diffusion model, the LLM, and search/retrieval now work as a single system, so a user can go from a product link to a try-on without manual steps in between.
• Infrastructure that scales with demand. Running on RunPod's edge network means our client isn't paying for idle GPU capacity between usage spikes, and can add capacity as usage grows.

Conclusion

This wasn't just a model swap — it was rebuilding the pipeline around how the product actually gets used: fast, on a phone, mid-shopping-decision. Fine-tuning CatVTON with LoRA/DoRA got the generation quality and cost where it needed to be, the LLM and search layers turned raw image generation into a real product feature, and edge deployment on RunPod made the whole thing fast enough to disappear into the background. That's what "try before you buy" needs to feel like: instant, not impressive-in-a-demo.