# Zero trust with Kafka

[Grab](https://yomu.fyi/company/grab) · Fabrice Harbulot · Dec 7, 2022

**Type:** Problem & solution

## Summary

Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.

## Context

Grab's Coban data platform team needed to transition large-scale Kafka clusters from pure network-based access control to a zero-trust model ensuring mutual authentication, least-privilege auditable authorization, and in-transit encryption.

## Approach / What changed

The team deployed Strimzi Kafka on Kubernetes configured with mTLS using short-lived certificates issued by HashiCorp Vault, integrated Open Policy Agent (OPA) for Policy-Based Access Control via GitOps pipelines, and updated their Go client SDK to manage in-memory certificate renewals.

## Takeaways

- mTLS was chosen over OAuth and SCRAM because it permits offline peer identity verification across disparate network environments without a live connection to a central auth authority.
- The Golang client SDK stores ephemeral certificates and private keys exclusively in memory and automatically initiates renewals at two-thirds of their TTL to eliminate CRL maintenance and reduce attack surfaces.
- Each Kafka cluster runs a dedicated Open Policy Agent deployment spanning matching Availability Zones, isolating the blast radius and allowing broker pods to cache authorization decisions locally.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Authentication](https://yomu.fyi/topic/authentication), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/zero-trust-with-kafka)