Lyft’s Feature Store: Architecture, Optimization, and Evolution

Summary

Lyft built a centralized Feature Store to optimize feature management, deployment, and data consistency for machine learning workflows across its rideshare platform. The system ingests batch features defined by Spark SQL and JSON metadata into Airflow DAGs, which simultaneously write data to Hive for model training and to an online serving layer for low-latency inference. Real-time streaming features are processed with Apache Flink and Apache Kafka before being ingested into the serving infrastructure. The online layer, dsfeatures, couples DynamoDB with a ValKey write-through LRU cache and OpenSearch to serve embeddings. Infrastructure optimizations reduced standard P95 read latency by a third while supporting over a trillion additional read and write operations.

Context

Lyft needed to centralize feature engineering and guarantee consistency across offline model training and real-time inference for dozens of critical rideshare workflows.

Approach / What changed

Lyft implemented a platform of platforms covering batch, streaming, and online serving. Batch features use Spark SQL and JSON metadata to generate Airflow DAGs feeding Hive and dsfeatures. Streaming pipelines ingest Kafka events through Apache Flink applications. The online layer uses DynamoDB backed by a ValKey write-through LRU cache, OpenSearch for embeddings, and Golang and Python SDKs.

Takeaways

• The batch ingestion pipeline automatically converts Spark SQL queries and JSON configuration files into production-ready Airflow DAGs that deliver data to Hive and online storage.
• The dsfeatures online serving layer utilizes DynamoDB as primary storage, an OpenSearch integration for embedding features, and a ValKey write-through LRU cache to minimize retrieval latency.
• Platform optimizations and cache TTL tuning reduced P95 read latency by one-third while total aggregate read/write activity increased by over one trillion operations.

Related reading

• Lyft · Feb 19, 2026

  Scaling Localization with AI at Lyft

  Lyft re-architected its batch localization pipeline to combine artificial intelligence with professional linguist oversight. The previous workflow relied exclusively on human translation, which resulted in multi-day turnarounds and linear cost scaling that hindered rapid market expansions. Under the new architecture, a dual-path system generates three candidate translations using a fast generation model, evaluates them against defined rubrics using a reasoning-focused model, and immediately releases approved drafts. Linguists review the drafts asynchronously in the Smartling translation management system to establish the final system of record. This pipeline reduced delivery latency from days to minutes while achieving a state where 95% of machine-generated translations require no significant changes during human review.

  Stefan Zier

• Grab · Jul 24, 2025

  The evolution of Grab's machine learning feature store