Zero trust with Kafka

Summary

Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.

Context

Grab's Coban data platform team needed to transition large-scale Kafka clusters from pure network-based access control to a zero-trust model ensuring mutual authentication, least-privilege auditable authorization, and in-transit encryption.

Approach / What changed

The team deployed Strimzi Kafka on Kubernetes configured with mTLS using short-lived certificates issued by HashiCorp Vault, integrated Open Policy Agent (OPA) for Policy-Based Access Control via GitOps pipelines, and updated their Go client SDK to manage in-memory certificate renewals.

Takeaways

• mTLS was chosen over OAuth and SCRAM because it permits offline peer identity verification across disparate network environments without a live connection to a central auth authority.
• The Golang client SDK stores ephemeral certificates and private keys exclusively in memory and automatically initiates renewals at two-thirds of their TTL to eliminate CRL maintenance and reduce attack surfaces.
• Each Kafka cluster runs a dedicated Open Policy Agent deployment spanning matching Availability Zones, isolating the blast radius and allowing broker pods to cache authorization decisions locally.

Related reading

• Grab · Apr 23, 2024

  Ensuring data reliability and observability in risk systems

  Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.

  Yi Ni Ong

• Grab · Jun 5, 2024

  Profile-guided optimisation (PGO) on Grab services

  Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval.

  Yonghao Hu

• Grab · May 3, 2024

  No version left behind: Our epic journey of GitLab upgrades

  Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.

  Saurabh Vajpayee

• Grab · Apr 3, 2024

  Iris - Turning observations into actionable insights for enhanced decision making

  Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations.

  Huong Vuong