Building Production-Scale Visual AI for Claims and Returns at Swiggy Instamart

Summary

Swiggy Instamart built two vision-language-model systems to handle more than 40,000 daily claims involving wrong, expired, poor-quality, damaged, and packaging-related items, while reducing reliance on human verification. The Image Validation Model (IVM) compares customer uploads with catalog images, routes each category through a specialised LangGraph StateGraph handler, and combines its structured verdict with fraud, confidence, and business-rule signals. Reverse Quality Check (RQC) compares delivery executives’ pickup photos with the original claim images, allowing matching returns to proceed and prompting retries when they do not. The architecture keeps visual assessment probabilistic but makes routing, aggregation, escalation, and financial decisions deterministic, with traces for category, outputs, confidence, cost, and latency. After more than six months in production, the systems are associated with a 6% customer-satisfaction increase, 20,000 fewer weekly Image Validation tickets, 98.5% RQC accuracy, and over 5,000 daily RQC tickets resolved without human intervention.

Takeaways

1. 1

   IVM treats the customer’s uploaded image as primary evidence; comments, product names, and categories provide context but cannot establish claims that images cannot verify.

2. 2

   IVM decomposes claim handling by category because wrong-item identity checks and expiry-date extraction require different visual criteria, failure modes, and output structures.

3. 3

   RQC reduced its False Not Match rate from 60% to 17% through iterative prompt work and automated optimisation, while the production system reports 98.5% accuracy.