How AWS CloudFormation Turns a Template Into a Reconciled Stack
CloudFormation is a declarative reconciler: you describe the target state in a template, and it provisions, updates, and reports on the stack that results. Here is the loop.
Loading…
Topic
AWS
199 posts about AWS. Every summary links to the original.
- 199 posts
- 29 companies
- 4 explainers
See the idea
AWS explainers
Amazon EVS Architecture: Where vSAN Ends and FSx for ONTAP Begins
Amazon EVS runs VMware Cloud Foundation on AWS bare metal, so the design question becomes where each workload's data lives. This layered diagram shows how vSAN, FSx for ONTAP, and cross-Region DR divide the work.
Amazon EKS as a Platform Boundary Map
See how a shared Amazon EKS architecture separates workloads, platform services, and network connectivity so teams can scale delivery without multiplying operational ownership.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How Grab Leveraged Performance Marketing Automation to Improve Conversion Rates by 30%
Grab faced operational bottlenecks managing direct-response Google Ads campaigns across thousands of ad groups due to its hyperlocal marketing across Southeast Asian markets. To eliminate the manual burden of tracking and updating ad creatives, the team built CARA, an in-house automation tool deployed on AWS serverless compute. CARA utilizes standardized file naming conventions to map assets to specific campaigns and connects with Google Ads and YouTube APIs to detect and replace low-performing assets. During an experimental rollout across more than 8,000 active ad groups, CARA replaced nearly 2,000 underperforming creatives. The automated asset replacement workflow produced an 18% to 30% increase in clickthrough and conversion rates.
Supabase Beta January 2021
Supabase released a collection of platform updates during January 2021 covering database tools, authentication integrations, regional availability, and infrastructure performance. Client libraries added support for PostgREST exact, planned, and estimated count queries to eliminate counting friction commonly experienced in platforms like Firebase. Authentication capabilities expanded to include Facebook and Azure login providers, an in-dashboard GoTrue audit trail, a helper function for user emails, and a prebuilt React Auth Widget. The platform also enabled new database hosting regions in London and Sydney while transitioning storage to GP3 for improved throughput. Furthermore, migrating API subdomains to Route53 alongside custom Let's Encrypt certificates yielded read performance benchmark gains of up to twelve percent.
Optimally Scaling Kafka Consumer Applications
Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.
Securing and Managing Multi-cloud Presto Clusters with Grab’s DataGateway
Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment.
Continuous PostgreSQL Backups using WAL-G
Continuous PostgreSQL backups require coordinating physical base snapshots with ongoing write-ahead log archiving to support point-in-time recovery. The open-source WAL-G utility simplifies this workflow by managing backup pushes and fetches to cloud storage services such as Amazon S3. In a typical setup on Ubuntu with PostgreSQL 12, envdir supplies AWS credentials and storage prefixes to WAL-G commands embedded directly in the PostgreSQL archive configuration. Regular base backups can be automated via cron jobs, which minimizes the volume of WAL archives that must be replayed during a restore. When recovering to a new instance, administrators fetch the latest base backup, define a restore command with targeted recovery timestamps in the configuration, and initiate replay using a recovery signal file.
The Journey of Deploying Apache Airflow at Grab
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
Alpha Launch Postmortem
An unexpected viral submission on Hacker News prompted sudden alpha traffic for Supabase, attracting 30,000 visitors and over 1,400 signups in one week. The surge strained initial infrastructure, which initially relied on a single server running Docker Compose for middleware and Digital Ocean for customer databases. System bottlenecks emerged rapidly, including hit limits on Digital Ocean droplet counts, frequent physical server faults, and a three-hour provisioning outage when Cloudflare reached its 1,000-subdomain limit. In response, the team migrated middleware to multiple globally load-balanced ECS clusters via AWS Global Accelerator. Supabase also transitioned 1,800 servers from Digital Ocean to AWS t3a instances, nearly halving database startup times from 90 seconds to 50 seconds.
A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform
Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.
A House of Cards: An Exploration of Security When Building Docker Containers
The post examines how a Docker-based, multi-tenant build environment can expose untrusted code when layered isolation is misconfigured. It traces a setup using AWS CodeBuild, Docker-in-Docker, and per-step build containers, then tests directory traversal through configurable Dockerfile paths before focusing on daemon networking. The vulnerability comes from starting dockerd with tcp://0.0.0.0:2375 and placing containers on a shared default network; build code can reach the daemon, issue Docker commands, mount the host filesystem, and retrieve container-managed AWS credentials. The excerpt reports a reverse shell, access to the CodeBuild filesystem, and temporary credentials whose impact depends on IAM permissions, while noting that the build must remain alive. The stated fix is to remove the TCP listener and retain the mapped Unix socket.
Dealing with the Meltdown Patch at Grab
AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.
Migrating Existing Datastores
Grab's Identity team faced imminent memory exhaustion on a single Redis node used to cache mobile authentication tokens under rapid user growth. Because read traffic outweighed write traffic by roughly 200 times, the team opted for an AWS ElastiCache cluster with three shards and two read replicas per shard. They executed a zero-downtime, six-phase migration plan while handling a peak load of 20,000 queries per second. The migration transitioned through initial one-time data replication, asynchronous shadow writes, synchronous dual writes, asynchronous read validation, switching primary reads, and final write cleanup. Controlled by feature flags and monitored with metrics at every stage, the migration completed without invalidating tokens or causing service disruptions.
DNS Resolution in Go and Cgo
Go applications experiencing load balancing issues across AWS Elastic Load Balancer (ELB) nodes trace uneven traffic distribution to IP address sorting defined in RFC 6724. Comparing Go's native DNS resolver with Cgo and glibc's getaddrinfo shows that both initially sort destination addresses using Rule 9 longest matching prefix rules. Disabling IPv6 on the network interface causes C and Cgo resolvers to return IP addresses in randomized order, while the native Go resolver continues deterministic sorting. Examination of net/addrselect.go reveals that Go's native resolver implements only a subset of the RFC rules and omits dynamic source address selection. Achieving permanent parity requires modifying the Go source code directly.
Driving Southeast Asia Forward with AWS
Grab transitioned its transportation platform from a single Ruby on Rails monolith on Amazon EC2 and Amazon RDS MySQL to a microservices architecture hosted on Amazon Web Services. The platform processes multi-petabyte real-time data flows and hundreds of millions of GPS data points to match drivers with passengers and push proactive demand heat maps. Operational efficiency is sustained with fewer than ten full-time infrastructure engineers, supported by AWS managed services. The analytics backend also transitioned from MySQL to Amazon Redshift, eventually moving to an Amazon S3 data lake using Amazon EMR and Presto. These data-driven matching systems improved driver-passenger allocation rates by up to 30%.
Troubleshooting Unusual AWS ELB 5XX Error
Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.
Scaling Like a Boss with Presto
Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.
A Key Expired in Redis, You Won't Believe What Happened Next
Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.
This Rocket Ain't Stopping - Achieving Zero Downtime for Rails to Golang API Migration
Grab transitioned its public passenger app APIs from a legacy Rails application to a Golang service-oriented architecture to consolidate its codebase and engineering teams. Initial attempts to proxy traffic through a cloned Rails server via gRPC were abandoned after encountering TCP load imbalances during autoscaling events and memory leaks in the gRPC Ruby gem. The team pivoted to direct logic migration, porting Ruby logic directly into Go while decomposing modules into standalone services. Verification relied on log-based load testing and live shadow testing, where write operations were safely validated using mock data access layers that evaluated expected database outcomes. Production rollout progressed endpoint-by-endpoint using requests-per-second traffic throttling and prewarmed AWS Elastic Load Balancers before executing the final DNS switch.
Round-robin in Distributed Systems
Building client-side load balancing for Grab's Common Data Service prompted a move from AWS Elastic Load Balancers to DNS discovery due to persistent connection issues and unpredictable scaling events. After patching an open-source library that failed to rotate IP sequences properly, the author evaluated different Go patterns for round-robin routing. A mutex-protected array counter provides the simplest model for basic retrieval, though adding mutations requires careful lock coordination. Alternatively, a dedicated balancer goroutine receiving requests over nested channels enables explicit operation timeouts and centralized event handling at the cost of higher code complexity and channel creation overhead. The author recommends the mutex approach for resource fetching and the goroutine-based design for workload balancing.
The Curious Case of the Phantom Instance
Datadog dashboards for the grab_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.