# Streaming > 36 posts about Streaming, summarised, each linking to the original. ## Articles ### [Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation](https://yomu.fyi/post/performance-bottlenecks-of-go-application-on-kubernetes-with-non-integ.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shubham Badkur - Published: May 23, 2023 Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above. ### [Safer deployment of streaming applications](https://yomu.fyi/post/safer-deployment-of-streaming-applications.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shi Kai Ng - Published: May 2, 2023 Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures. ### [Migrating from Role to Attribute-based Access Control](https://yomu.fyi/post/migrating-from-role-to-attribute-based-access-control.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Minh Khoi Nguyen - Published: Mar 9, 2023 Grab's streaming data platform team migrated the Kafka Control Plane from Role-Based Access Control to Attribute-Based Access Control to eliminate operational bottlenecks and manual permission management. The previous model required defining hundreds of roles, permissions, and group mappings in an internal IAM service, leading to approval delays and stale memberships. Under the new architecture, user attributes sync from the HRMS and token payloads, while resource attributes are tagged upon creation or backfilled to reflect department and team ownership. Open Policy Agent evaluates access requests defined in Rego via middleware by comparing user attributes with resource metadata. This transition eliminated over 200 roles, 200 permissions, and roughly 3,000 unused IAM resources while automating access provisioning for new joiners. ### [Securing GitOps pipelines](https://yomu.fyi/post/securing-gitops-pipelines.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Thang Le - Published: Mar 1, 2023 Grab's real-time data platform team transitioned from an Atlantis-driven Terraform workflow to an in-house GitOps platform called Khone to manage streaming infrastructure resources like Kafka topics and Flink pipelines. The earlier setup suffered from coarse-grained access controls, required manual merge request comments, and lacked flexible validation capabilities within native configuration files. Khone derives environment parameters directly from standardized directory paths and uses Python with the python-hcl2 library to inspect and validate resource definitions before executing Terraform stages in parallel. To prevent configuration tampering in merge requests, CI/CD pipeline definitions and execution scripts are isolated in a separate administrative repository and fetched during job runs using shallow Git clones. ### [Zero trust with Kafka](https://yomu.fyi/post/zero-trust-with-kafka.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Fabrice Harbulot - Published: Dec 7, 2022 Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput. ### [Automatic rule backtesting with large quantities of data](https://yomu.fyi/post/automatic-rule-backtesting-with-large-quantities-of-data.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chao Wang - Published: Sep 8, 2022 Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes. ### [How Kafka Connect helps move data seamlessly](https://yomu.fyi/post/how-kafka-connect-helps-move-data-seamlessly.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Wenli Wan - Published: Apr 6, 2022 Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery. ### [Real-time data ingestion in Grab](https://yomu.fyi/post/real-time-data-ingestion-in-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shuguang Xiang - Published: Mar 14, 2022 Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails. ### [Exposing a Kafka Cluster via a VPC Endpoint Service](https://yomu.fyi/post/exposing-a-kafka-cluster-via-a-vpc-endpoint-service.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Fabrice Harbulot - Published: Feb 18, 2022 To replace VPC peering and reduce attack surfaces, Grab exposed a multi-Availability Zone Apache Kafka cluster in its main AWS VPC to clients in a separate GrabKios VPC using AWS VPC Endpoint Service. Because Kafka requires clients to establish deterministic connections to individual brokers, the team configured a Network Load Balancer with unique TCP ports and dedicated target groups for each broker alongside a shared bootstrap port. They added custom listeners on the Kafka brokers to advertise endpoints using private Route 53 CNAMEs rather than raw interface hostnames. To eliminate unnecessary cross-AZ network latency and data transfer costs, the architecture was refined to advertise AZ-specific private CNAMEs mapped directly to zonal endpoint interfaces. ### [Automating Multi-Armed Bandit testing during feature rollout](https://yomu.fyi/post/automating-multi-armed-bandit-testing-during-feature-rollout.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Weicheng Zhu - Published: Sep 1, 2021 Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic. ### [Optimally Scaling Kafka Consumer Applications](https://yomu.fyi/post/optimally-scaling-kafka-consumer-applications.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shubham Badkur - Published: Oct 13, 2020 Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays. ### [Plumbing At Scale](https://yomu.fyi/post/plumbing-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Karan Kamath - Published: Jan 6, 2020 Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling. ### [Using Grab’s Trust Counter Service to Detect Fraud Successfully](https://yomu.fyi/post/using-grab-s-trust-counter-service-to-detect-fraud-successfully.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chao Wang - Published: Oct 21, 2019 Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs. ### [Data First, SLA Always](https://yomu.fyi/post/data-first-sla-always.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Johan Kok - Published: Aug 1, 2019 Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches. ### [How We Simplified Our Data Ingestion & Transformation Process](https://yomu.fyi/post/how-we-simplified-our-data-ingestion-transformation-process.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yichao Wang - Published: Mar 3, 2019 Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute. ### [Querying Big Data in Real-time with Presto & Grab's TalariaDB](https://yomu.fyi/post/querying-big-data-in-real-time-with-presto-grab-s-talariadb.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Jan 2, 2019 Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second. [Newer posts](https://yomu.fyi/topic/streaming.md)