The evolution of Grab's machine learning feature store

Summary

Grab describes how it evolved its machine learning feature store as increasingly complex models began requiring high-cardinality keys and high-dimensional data. It replaced its initial Amphawa approach with feature tables: data-lake tables whose contents are atomically ingested into serving storage, with emphasis on consistency and reproducibility. The workflow stores feature tables and version-control metadata as Parquet objects in Amazon S3, validates schemas, loads data through reverse ETL into Aurora Postgres, and warms new tables through shadow reads before switching. The redesign supports contextual retrieval, atomic collection-wide updates, and isolation between reading and writing capacity; its SDK remains a thin database-driver wrapper that translates queries to SQL. Grab also chose decentralised deployments for version flexibility, rollout strategies, noisy-neighbor mitigation, and improved caching, concluding that Amazon Aurora for Postgres provides a robust, scalable, cost-effective foundation.

Takeaways

1. 1

   Feature tables are ingested atomically, so feature changes become visible only after the entire operation succeeds, reducing compatibility concerns during model or data-version changes.

2. 2

   The ingestion workflow uses Amazon S3 for Parquet-based feature tables and metadata, then loads data into Aurora Postgres through a reverse ETL process with pre-ingestion validation and post-ingestion warm-up.

3. 3

   Decentralised data-serving deployments were selected because they support blue-green and rolling deployments, multiple coexisting versions, easier rollbacks, service isolation, and improved caching efficiency.