How MakeMyTrip achieved millisecond personalization at scale with Databricks

Summary

MakeMyTrip needed sub-second personalization for “last-searched” hotels across its consumer and corporate travel businesses, serving millions of daily users. Spark Structured Streaming’s micro-batch mode consistently delivered one to two seconds, while adopting Flink would have added a second engine, duplicated logic and increased operational overhead. The team instead adopted Databricks Real-Time Mode (RTM), using continuous data flow, concurrent pipeline scheduling and streaming shuffle to process events with millisecond-level latency on Spark. Its unified pipeline merges B2C and B2B Kafka streams, enriches events with each user’s “Last N” hotels from Aerospike, and writes results to Redis for serving in under 50ms. After tuning partitions, backpressure and rate limits, the production deployment reduced P50 latency from approximately 1.23 seconds to 44ms, P99 latency from over one minute to about 500ms, and increased click-through rate by 7%.

Takeaways

1. 1

   RTM replaced Spark’s periodic micro-batches with continuous data flow, concurrent stage scheduling and streaming shuffle, removing latency sources that kept micro-batch processing at one to two seconds.

2. 2

   A single unified pipeline applies the same personalization logic to B2C and B2B streams, avoiding the duplicated code and potential logic drift of separate Spark and Flink systems.

3. 3

   Production tuning reduced P50 latency to 44ms and P99 latency to about 500ms; click-through rate increased by 7%, while Redis enabled personalized results to be served in under 50ms.

FAQ