Building a Magic Mirror: AI retail experiences with Remix

Summary

Shopify’s “magic mirror” is an AI-powered retail installation that uses a display and hidden webcam to deliver personalized messages, animations, and product interactions in under a minute. A Remix route handles the browser and server boundary: a button-triggered POST captures an image, sends it with a curated prompt to a vision-capable language model, and renders a compliment plus a makeup recommendation; server-sent events optionally let a remote device trigger or reset the experience. The prompt combines persona and safety constraints, a plain-text product and inventory table, and a required JSON schema, while strict product-ID lookup provides graceful degradation. The interface uses explicit states, delayed capture, timed response and reset screens, and animated transitions. The production setup keeps brand configuration, shared components, actions, and SSE infrastructure reusable across mirror modes.

Context

As online shopping becomes frictionless, physical retail is described as shifting toward interactive, surprising, and personalized experiences. The mirror concept was first applied to Rare Beauty makeup shade matching, where guests could receive a blush recommendation and sync metadata to their customer profile.

Approach / What changed

The implementation uses a full-stack Remix route with a server action for image processing and a browser component for the experience. A vision-capable model receives the captured image and a prompt containing persona rules, safety exclusions, product and inventory data, and a JSON output schema. The interface manages capture, loading, response, countdown, and exit states, while optional server-sent events connect remote controls to individual mirror instances.

Takeaways

• A deliberate two-second delay between the trigger and screenshot gave customers time to reposition; testing found that centered, arm’s-length photos produced better skin tone analysis than close-up, off-angle shots.
• The prompt uses concrete limits, a curated compliment list, explicit exclusions, and inventory deltas. The source reports that character limits and curated options improved consistency, while exclusions were treated as essential for safety.
• Server-sent events allow a separate device to trigger captures or reset mirrors without polling. Channel names let multiple mirror instances share one server while ignoring events intended for other booths.

Related reading