Loading…
Grab
Southeast Asia's leading everyday "super-app" that provides ride-hailing, food delivery, grocery shopping, and digital financial services.
- 242 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Grab's service mesh evolution: From Consul to Istio
Grab describes migrating its service mesh from Consul and the Catcher fallback across a microservices ecosystem of more than 1,000 services spanning VMs, EKS clusters, AWS, and GCP. The existing setup exposed fleet-wide risk from a single Consul server issue, added fallback complexity, and lacked advanced traffic management, security controls, and robust multi-cluster support. After evaluating Application Load Balancer, Istio, AWS Lattice, and Linkerd for stability, performance, integration, and operational simplicity, the team selected Istio for its Kubernetes integration, flexible deployment models, and feature set. The design uses multiple control planes on dedicated Kubernetes clusters, active-active pairs, organization-based mesh boundaries, custom resource definitions, and cross-cluster endpoint discovery to support isolation, availability, routing, security, and observability. Migration began in Q4 2024 with GCP-to-AWS work alongside gRPC and HTTP traffic initiatives, using gradual shifting, rollback mechanisms, and performance monitoring while teams manage legacy systems and cross-mesh connectivity.
DispatchGym: Grab’s reinforcement learning research framework
DispatchGym is a research framework for reinforcement learning (RL) studies in dispatch systems that match bookings with drivers. It combines an RL algorithm, a dispatch-process simulation, and a Gymnasium API interface, allowing data scientists with basic RL and Python skills to modify and test ideas independently. The simulated environment models booking creation, driver dispatch and movement, and booking completion, prioritizing directional accuracy over exact metric matching; its modular Python architecture uses Numba to reduce runtime overhead. A command-line upload launches the modified code on internal infrastructure as a Spark job, avoiding infrastructure management while preserving flexibility across simulation and optimization components. The framework supported experiments with contextual bandits, including linear, neural-linear, and Gaussian-process variants alongside several action-sampling strategies, to identify combinations of levers and reward functions for dispatch hyperparameter tuning and improved fulfilment efficiency and reliability.
Counter Service: How we rewrote it in Rust
The Integrity Data Platform team rewrote Counter Service, a high-QPS Golang microservice that consumes streams, writes counts to Scylla, and exposes gRPC queries backed by Scylla and Redis. They selected it because BatchRead reached tens of thousands of QPS while remaining constrained in functionality, making infrastructure savings potentially worth the rewrite effort and Rust learning curve. Rather than translating code line by line, the team treated the existing service as a black box, preserved its interfaces and contracts, and rebuilt the application logic from scratch; it also assessed Rust ecosystem support for dependencies including tonic, Actix Web, Fred, and the Scylla driver. The rewrite delivered similar performance with 70% infrastructure savings, according to the abstract. The conclusion reports the service as 80% cheaper, endorses Rust for new services, and cautions that async Rust is challenging.
The complete stream processing journey on FlinkSQL
The post describes replacing per-user Zeppelin notebooks with a shared FlinkSQL gateway to improve interactive stream processing over Kafka data and support production pipelines. Zeppelin notebooks were limited by support for Flink 1.17, roughly five-minute cold starts, and isolation from internal platforms, while current Flink was 1.20. The replacement uses compute, integration, and query layers: a shared FlinkSQL gateway submits SQL through its REST API to session-cluster jobs, while a custom control plane handles authentication, authorization, session management, orchestration, and paginated results for UI and headless API clients. An interactive UI uses a Hive Metastore catalog to expose Kafka topics as tables, and the platform returns results in one minute rather than five; a configuration-based deployment tool provisions resources and deploys SQL-defined Flink pipelines within ten minutes. After release, the post reports an uptick in created pipelines and interactive queries, but it does not quantify that increase.
Effortless enterprise authentication at Grab: Dex in action
Grab built Concedo to centralise access management, authentication, and authorisation across internal applications and external platforms. Its customised Google OAuth2.0-based mechanism created integration challenges with systems such as Databricks and Datadog, producing fragmented sign-on experiences and administrative overhead. To standardise identity, the team adopted OpenID Connect (OIDC) and used Dex, a federated OpenID Connect provider that aggregates identity providers and issues standardised OIDC tokens. Dex supports single sign-on, multiple identity providers, token delegation between trusted peer services, and a kill switch that can redirect traffic during an identity provider outage. The implementation separated application authentication concerns and established a single identity source, while a future phase will address fragmented authorisation through a unified model.
From failure to success: The birth of GrabGPT, Grab’s internal ChatGPT
In March 2023, an effort to reduce repetitive questions overwhelming Grab’s ML Platform support channels began as a chatbot for platform documentation and became GrabGPT, an internal ChatGPT-like tool. The initial chatbot used chatbot-ui and more than 20,000 words of Q&A documentation, but GPT-3.5-turbo’s 8,000-token limit forced summaries below 800 words; embedding search also performed poorly, so the approach was abandoned. The pivot extended existing frameworks, added Google login, and deployed a tool through Grab’s catwalk model-serving platform, later named GrabGPT. It reached 300 registered users on day one, more than 3,000 users and 600 daily active users by month three, and nearly all Grabbers according to the post, with private routing, global accessibility, multi-provider model support, and auditable interactions cited as reasons for adoption.
Streamlining RiskOps with the SOP agent framework
Risk Operations teams investigating Account Takeover (ATO) cases must gather data across systems, run SQL queries, and make decisions under pressure, creating delays and error risk. The SOP-driven LLM agent framework represents the workflow as a natural-language, indentation-based tree whose steps specify queries, external functions, and decision criteria. During execution, the SOP planner navigates the tree and produces JSON-formatted instructions, while a Worker Agent invokes APIs or SQL queries and stores results in a feedback loop. The process ends by synthesising investigation data into a report and final decision. In reported results, handling time fell from 22 to 3 minutes per ticket, 87% of ATO cases were automated since launch, and the error rate was zero.
Introducing the SOP-driven LLM agent frameworks
Grab introduces an LLM agent framework that uses Standard Operating Procedures (SOPs) to guide enterprise automation and improve consistency in complex processes. It represents SOPs as trees of action and decision nodes, while a planner uses Depth-First Search, prompt generation, and backtracking to direct an LLM-powered worker agent; context compression and limited APIs are used to address hallucination and efficiency concerns. An SOP editor, plugins for APIs, Python, and SQL, workflows, a GRAG pipeline, playground, multilingual user agent, and state stack extend the framework’s capabilities. Reported evaluations include over 99.8% accuracy, an Account Takeover Investigations bot automating 87% of cases and reducing investigation time from 23 minutes to three, and fraud investigations cutting AHT by 45% and saving over 300 man-hours monthly.
Evaluating performance impact of removing Redis-cache from a Scylla-backed service
A Rust service at Grab serves tens of thousands of peak QPS for batched count requests, using Scylla, Redis, and MySQL counter configurations. Its 15-minute cache-key truncation and five-minute TTL caused cache-hit declines and recurring Scylla QPS spikes at the end of each quarter-hour, when recent-data requests generated new keys. The experiment progressively disabled Redis, while a daily major compaction job was added to Scylla nodes; the service continued combining minutely, hourly, and daily tables for range calculations. With Redis fully decommissioned, Scylla cache hits rose from about 30% to 70% of reads, cache misses stayed about the same despite doubled traffic, peak CPU moved from roughly 50% to 52%, and overall service performance was maintained while eliminating a component responsible for about 25% of service cost.
Facilitating Docs-as-Code implementation for users unfamiliar with Markdown
Grab introduced a rich-text, WYSIWYG editor in its TechDocs portal to help non-engineering contributors work within a Docs-as-Code process despite limited familiarity with Markdown and GitLab. The editor keeps GitLab as the central storage system, uses Lexical to import and export Markdown, and supports page reordering, renaming, deletion, formatting, diagrams, and switching between normal and Markdown views. Three user journeys cover editing existing TechDocs and creating stand-alone or entity-linked documentation, with GitLab OAuth2, merge-request creation, repository access checks, and scheduler-based entity-catalog registration after merges. A phased rollout added drafts, image uploads, diagram integrations, concurrent-session visibility, and live preview, while pending-MR displays and conflict-resolution tools address collaboration risks.
Improving Hugo stability and addressing oncall challenges through automation
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker_uuid and Spark event metrics through app_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.
TechDocs at Grab: Cultivating a culture of quality documentation
Grab’s TechDocs working group describes a two- to three-year effort to make high-quality documentation part of everyday engineering culture. Built on the Helix engineering platform and a Docs-as-Code workflow, TechDocs gives teams a central place to create, manage, and access technical content, while user research informs tool and process changes. The programme combines inventory-taking, policy, team enablement, and recurring measurement, using broad surveys and 1:1 observation to identify fragmented tools, weak ownership, search problems, and maintenance gaps. Research found that only 2% of engineers reported resolving 80–100% of on-call questions through documentation; later initiatives included an editor, writing training, practical workshops, and recognition tied partly to documentation quality and on-call-bot response accuracy. Planned work includes better navigation, training, feedback loops, and editing workflows.
Grab AI Gateway: Connecting Grabbers to multiple GenAI providers
Grab AI Gateway is a central access layer connecting Grabbers’ applications to multiple GenAI providers, including OpenAI, Azure, AWS Bedrock, and Google VertexAI. Built as a set of reverse proxies, it centralizes provider authentication, authorization, rate limiting, auditing, and capacity management while presenting a unified OpenAI-style API; users can switch models by changing the model parameter. Exploration keys support short-lived staging experiments, while reviewed use cases can use longer-term service keys, and the gateway exposes embedding, image, audio, fine-tuning, search, and in-house open source models. Shared capacity, dynamic routing, monitoring, cost attribution, and usage records support governance and efficiency, although batch traffic can interfere with latency-sensitive services and provider-specific SDK edge cases require integration testing. More than 3,000 Grabbers have requested exploration keys, and the gateway powers applications including ride-safety audio analysis, content moderation, SQL generation, incident automation, and Slack support.
Embracing passwordless authentication with Grab’s Passkey
Grab’s Passkey adds passwordless authentication to its app to address phishing, keystroke-logging, brute-force, and other security threats associated with passwords plus the friction of OTP-based login. Built on the FIDO standard and implemented through WebAuthn, the flow uses a backend, frontend, and authenticator: the authenticator creates and stores a passkey, while Grab stores the public key and metadata. Registration uses a server-generated challenge and parameters including rp.id, user information, resident-key requirements, and supported public-key algorithms; sign-in returns a credential whose signature is verified against the stored public key. Google Password Manager and iCloud Keychain can sync passkeys across supported devices, and users authenticate with their screen lock rather than entering a password or waiting for an OTP. The implementation is described as faster, smoother, phishing-resistant login with potential savings on third-party authentication and messaging costs, while Grab plans to evaluate web login expansion through adoption, feedback, and usage data.
Turbocharging GrabUnlimited with Temporal
Grab’s GrabUnlimited flagship membership program struggled as subscribers grew by over 1000% from January 2022 to June 2023. Its original design used a daily cron job for renewals, Amazon’s SQS to distribute state handlers, and five-minute Redis locks, but outages, corrupted states, concurrent cancellation and renewal, missed benefits, and non-idempotent retries created operational problems. The team migrated the service to a Temporal workflow-based system, enhancing its ability to handle millions of subscribers with greater efficiency and resilience. The source reports an 80% reduction in open production incidents and says the change improved the membership experience for users. This first part focuses on designing a robust and scalable membership system, while migration challenges and lessons are reserved for a subsequent installment.
How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication
Grab migrated its stream-processing (write) functionality from an existing service into a separate service while handling 20,000 reads per second from 16 Kafka streams and writing to output streams and DynamoDB tables. The migration required gradual stream-by-stream rollout, production validation, and a no-downtime switchover; feature flags were rejected because deployment timing could cause loss, duplication, or inconsistent values. The team moved processing into shared /commons code, created temporary validation sinks, and scheduled each switchover with a shared map[string]time.Time, allowing both services to change writers at the same time. During each cutover, both services consumed streams, the old service stopped writing to resources, and monitoring continued before old consumption was removed; the result was zero data loss or duplication, no downtime, and completion in about three weeks. It was temporary and tested, but rolling back either service could create inconsistency if systems fell out of sync.
Supercharging LLM application development with LLM-Kit
Grab developed LLM-Kit, a comprehensive framework for setting up production-ready Generative AI applications while addressing security, scalability, integration, monitoring, and standardisation concerns. Its pre-configured structure includes an API server, configuration management, a sample LLM Agent, tests, and an integrated stack spanning Poetry, Gunicorn, FastAPI, LangChain, LangSmith, HashiCorp Vault, Amazon EKS, and GitLab CI. Engineers submit application details through a form, which triggers GitLab project and code-scaffold generation, CI/CD setup, Terraform-based staging and production infrastructure provisioning, and secure GPT-token storage in Vault. The scaffold also includes authentication, API documentation, observability, Redis and vector-database integrations, deployment pipelines, and a cookbook with organisational examples; the post reports that it has onboarded hundreds of GenAI applications and saves an estimated 1.5 weeks before teams begin their first feature.
How we reduced initialisation time of Product Configuration Management SDK
GrabX, Grab’s central product configuration management platform, provides an SDK whose eventually consistent updates take about a minute to reach clients, but initialisation took roughly four minutes for services loading around 400 MB. The SDK’s sequential workflow downloaded subscribed-service and common data from AWS S3 to disk and memory before initialising dependent modules, creating cold-start pressure and possible throttling when new instances were launched. Phase 1 made the two data loads concurrent, reducing initialisation time by approximately 80%, while Phase 2 concurrently downloaded and loaded large configurations within a subscribed service, adding a further approximate 6% reduction. Phase 3 removed disk persistence because its stale-data fallback conflicted with most clients’ need for current data, allowing failed connections to trigger client action such as retrying. Benchmarks across services with varied data sizes indicated reductions of up to 90%.
Metasense V2: Enhancing, improving and productionisation of LLM powered data governance
Metasense V2 addresses automated data governance by improving an LLM-powered metadata generation and classification system. It builds on an initial rollout that scanned more than 20,000 data entries at 300–400 entities per day, but retained human verification to catch misclassifications. Post-rollout owner feedback and manual classifications from the Data Governance Office supplied training and testing data, exposing failures involving PII in business email columns, nested JSON, and passenger communications. The revised approach splits PII and non-PII tagging, reduces the PII tag set from 21 to 8, shortens the prompt from 1,254 to 737 words, and divides tables exceeding 150 columns. LangChain and LangSmith support prompt development, evaluation across accuracy, latency, and error rate, deployment, and monitoring, while threshold-based alerts trigger a model-improvement protocol when misclassification rates rise.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.