Chimera Sandbox: A scalable experimentation and development platform for Notebook services

Summary

Grab's Chimera team built Chimera Sandbox to address resource accessibility challenges and enable rapid machine learning experimentation across technical and non-technical teams. The platform runs on multi-cluster Kubernetes infrastructure capable of scaling to thousands of nodes per cluster during peak loads. Users can provision customizable Docker environments with varying CPU and GPU configurations, collaborate in real time, and query data using Spark-on-Kubernetes integrations and helper magic cells. The workspace integrates with Grab's AI Gateway for LLM access alongside tools like MLflow, Optuna, RStudio, and LabelStudio on shared storage, backed by user-level cost attribution.

Context

Rapid machine learning and Generative AI experimentation across Grab was hindered by limited accessibility to compute resources like GPU instances and specialized services for building LLM-powered applications.

Approach / What changed

Grab built Chimera Sandbox, a multi-tenant workspace platform on Kubernetes clusters supporting custom Docker images, integrated Spark engines, an AI Gallery of prebuilt notebooks, and unified access to internal ML tools and LLM gateways.

Takeaways

• The compute infrastructure runs across multiple Kubernetes clusters that can scale up to thousands of nodes per cluster during peak usage.
• Integrated Spark tooling includes a SparkConnect Kernel, a custom Spark-on-Kubernetes operator, and a %%spark_sql magic cell for direct querying without manual session initialization.
• Each multi-tenant workspace includes user-level cost showback and chargeback attribution to track expenditure on GPU and compute instances.

Related reading

• Grab · Nov 14, 2024

  Metasense V2: Enhancing, improving and productionisation of LLM powered data governance

  Grab scaled Metasense to automate metadata generation, column-level classification, and sensitivity tiering across its entire data lake. Post-rollout analysis revealed that high tag volumes, wide tables, and mixed content—such as nested JSON and customer communications—strained model capacity and led to missed Personally Identifiable Information (PII). To overcome these limitations, the team split the classification process into distinct PII and non-PII tasks, reduced prompt word counts, and partitioned tables with over 150 columns into smaller units. Integrating LangChain and LangSmith modernized the architecture by enabling direct prompt experimentation, custom metric tracking, and version-controlled deployments. The updated pipeline achieved low misclassification rates alongside automated alert thresholds to trigger model improvement protocols if errors increase.

  Nick Buhrer

• Grab · Oct 30, 2024

  How we reduced peak memory and CPU usage of the product configuration management SDK

  Grab's central product configuration management platform, GrabX, previously aggregated all configurations across every backend service into a single JSON file hosted on AWS S3. Every minute, client SDKs fetched, parsed, and loaded this growing file—which exceeded 100MB—causing CPU throttling spikes, elevated P99 latency, and unnecessary memory consumption. Analysis revealed that 98% of services required less than 1% of the total configuration data. To resolve these bottlenecks, the team partitioned data by service, split configurations into separate S3 files under distinct prefixes, and introduced a per-service changelog for incremental updates. Benchmarks showed the redesign decreased maximum CPU utilisation by over 50% and reduced memory usage by up to 70%.

  Ram Dilip Pradhan

• Grab · Dec 12, 2024

  Turbocharging GrabUnlimited with Temporal

  GrabUnlimited experienced scaling bottlenecks, corrupted membership states, and elevated production incidents after its subscriber base grew by over 1000%. The original architecture relied on Amazon SQS state machines, 5-minute Redis locks, and daily batch cron jobs that overwhelmed the database and lacked granular idempotency during upstream retries. To eliminate these failure modes, the engineering team migrated the core membership lifecycle to Temporal's workflow orchestration engine. Replacing batch cron jobs with Temporal Timers distributed renewal operations throughout the day, while matching workflow IDs prevented race conditions between renewals and cancellations. This architectural transition resolved database bottlenecks and yielded an 80% reduction in open production incidents.

  Michel Parreno

• Grab · Oct 23, 2024

  LLM-assisted vector similarity search

  Vector similarity search often struggles with conceptual nuances, negations, and complex constraints despite its efficiency in high-dimensional semantic retrieval. To address this limitation, a two-step retrieval pipeline combines Facebook AI Similarity Search (FAISS) using OpenAI's text-embedding-ada-002 embeddings with a large language model re-ranking step powered by GPT-4o. The initial FAISS search extracts a candidate shortlist of matches, which GPT-4o then re-ranks based on contextual relevance and query intent. Evaluations on structured datasets showed that while simple queries yielded comparable results across methods, the LLM-assisted approach effectively resolved negations and conceptual matching where raw vector search failed. Internal production trials on relational datasets exceeding 4,500 rows demonstrated similar relevance improvements, with additional query latency being the primary operational trade-off.

  Md Riyadh