How AWS CloudFormation Turns a Template Into a Reconciled Stack
CloudFormation is a declarative reconciler: you describe the target state in a template, and it provisions, updates, and reports on the stack that results. Here is the loop.
Loading…
Topic
AWS
199 posts about AWS. Every summary links to the original.
- 199 posts
- 29 companies
- 4 explainers
See the idea
AWS explainers
Amazon EVS Architecture: Where vSAN Ends and FSx for ONTAP Begins
Amazon EVS runs VMware Cloud Foundation on AWS bare metal, so the design question becomes where each workload's data lives. This layered diagram shows how vSAN, FSx for ONTAP, and cross-Region DR divide the work.
Amazon EKS as a Platform Boundary Map
See how a shared Amazon EKS architecture separates workloads, platform services, and network connectivity so teams can scale delivery without multiplying operational ownership.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Growing your Stripe integration With Event Destinations
Stripe Event Destinations, now in general availability and rolling out to all accounts, lets companies extend payment workflows without changing existing integrations. The post explains how Stripe events can be sent either to an AWS account through Amazon EventBridge or to a webhook endpoint, with EventBridge routing selected events to services such as AWS Lambda, Step Functions, and SQS. Event filtering, logging, transformation, and built-in security reduce the need to operate webhook servers and verify signatures in application code. Serverless Lambda functions can add custom logic and connect to DynamoDB, S3, or SNS, supporting examples including date-limited memberships, partial student payments, and real-time tipping broadcasts through AWS IoT Core and WebSockets. The resulting architecture is presented as flexible and scalable for variable or growing event volumes while preserving the underlying Stripe integration.
How we reduced peak memory and CPU usage of the product configuration management SDK
GrabX’s product configuration SDK used substantial memory and CPU because every client downloaded, decoded, and loaded a single JSON file containing configurations for all services, even when it needed only a small subset. The design also regenerated and fetched complete files every minute, while decoding a file larger than 100 MB could starve request handling and increase P99 latency. The proposed redesign partitions data by service, stores each configuration in its own JSON file and S3 prefix, and adds service-level changelogs so SDKs subscribe only to required services and apply targeted updates. Benchmarking across varied configuration sizes found potential reductions of up to 70% in memory utilisation and more than 50% in maximum CPU utilisation. Further work is planned to reduce SDK initialisation time.
How to Create a Secure CI/CD Pipeline Using Okta Terraform
The guide presents a secure CI/CD pipeline for managing Okta resources with Terraform, AWS, and GitHub Actions, addressing local setup, collaboration, state storage, secrets, and deployment automation. Its architecture uses GitHub as source control and workflow trigger, GitHub OIDC to authenticate to AWS IAM STS and assume roles, S3 for Terraform state, DynamoDB for state locking, and AWS Secrets Manager for Okta OAuth 2.0 client credentials private keys. Developers push changes to develop for an automatic development build, open a pull request to main for a production plan, and merge approved changes to trigger the production build. The workflows run Terraform init, plan, and apply as appropriate, while the guide also covers repository setup, branch organization, and troubleshooting for missed configuration in GitHub, Okta, or AWS.
Unveiling the process: The creation of our powerful campaign builder
Grab details the event processing architecture behind Trident, its internal marketing campaign platform that evaluates If This, Then That (IFTTT) logic over Kafka streams. The core processing unit is a treatment consisting of an event, optional conditions, and actions. Complex campaign capabilities—such as counters, limits, and delays across multiple hours via recursive SQS message scheduling—are assembled from multiple coordinated treatments. To simplify campaign creation, Grab introduced a flowchart-like visual builder represented as a JSON node tree that compiles recursively into treatments while persisting node-to-treatment mappings to reconcile edits over time.
Bringing your Stripe objects to life with Workbench
The post demonstrates how Stripe Workbench’s Inspector can be used to examine a PaymentIntent throughout its lifecycle and diagnose payment workflow issues. It maps states including requires_payment_method, processing, requires_action, succeeded, and canceled, showing how the status property changes as inputs and decisions move the object through a state machine. The walkthrough creates and confirms test PaymentIntents with Stripe Shell commands, including a 3DS-required card that produces a requires_action state and a use_stripe_sdk next_action. It also describes reacting to state changes through webhook endpoints or Amazon EventBridge, using the Inspector to identify friction such as PaymentIntents stalled at requires_action and to debug abandoned payments.
AWS without Access Keys
The post describes how Fly Machines can access AWS resources such as an S3 bucket without distributing long-lived AWS access keys. It uses OpenID Connect (OIDC): AWS IAM trusts Fly.io’s OIDC identity provider, a role grants narrowly scoped permissions, and the app receives the role ARN as an environment variable. At boot, Fly’s Rust init requests a machine-specific OIDC token through the /.fly/api Unix socket, writes it to /.fly/oidc_token, and sets AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_SESSION_NAME so the AWS SDK can call STS AssumeRoleWithWebIdentity. AWS verifies the token, issues short-lived credentials, and evaluates the role’s IAM policy before serving the requested object. The trust policy can constrain tokens by audience and subject, including an organization, Fly App, or machine; the same pattern can authenticate against other OIDC-compliant providers, although Azure and GCP lack equivalent init integration.
Event Driven Machines
Event Driven Machines presents Lambdo, a pattern for using Fly Machines as short-lived, event-triggered compute rather than relying on HTTP activation. It polls an SQS queue for JSON events, batches the messages and metadata, then creates fast-booting Fly Machines with the events written to /tmp/events.json; the Machine is destroyed after processing finishes. Application code can come from an existing codebase or Lambdo’s Node and PHP runtime images, with handlers or commands processing each event. SQS message attributes select the required image and can specify VM size and command, while the approach supports arbitrary code and even GPU-enabled Machines, giving each event control over its execution environment.
Kafka on Kubernetes: Reloaded for fault tolerance
Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.
Graph modelling guidelines
Graph modelling uses graph theory to represent real-world entities, relationships, and properties through nodes and edges, unlocking insights across interconnected datasets. The process begins with defining domain concepts, mapping entities and relationships, and assigning relevant descriptive properties to both nodes and edges. Practitioners choose between graph structures such as property graphs or Resource Description Framework models before optionally developing a structural schema. Data is then imported or generated to populate storage systems such as Neo4j, Amazon Neptune, Azure Cosmos DB, or in-memory data structures. Finally, the graph is visualised and analysed using algorithmic methods like shortest path calculation, centrality measurement, and community detection.
Finding the right balance of speed and security through just-in-time access to cloud resources
Ramp’s rapid growth exposed problems in its AWS access model: three broad roles created persistent production access for some senior engineers while junior engineers depended on seniors for queries. The team first reorganized permissions into more than 20 team-specific AWS Identity Center groups, each backed by permission sets and a Terraform module that presents IAM configuration through human-readable resource lists. It then introduced just-in-time access through ConductorOne by pairing each non-production role with an elevated production role, removing standing production access and provisioning entitlements through AWS SSO group membership. Team-owned requests were automatically approved, while cross-team requests went to the relevant tech lead, with temporary access and backup approval handling described in the design. The resulting model reduced exposure, preserved fast access to needed resources, and recorded production activity for accountability and audit purposes.
How Ramp Accelerated Machine Learning Development to Simplify Finance
Ramp describes how its machine-learning organization used Metaflow to shorten the path from prototype to production for models spanning credit risk, fraud, growth, product, and Ramp Intelligence. Its initial riskiness model exposed long vendor-managed jobs, weak logging, limited Docker support, and substantial platform friction; the model took months to build. Metaflow, with developer-experience improvements, lets teams define flows in Python, run them locally or on AWS Batch, visualize results with cards, and connect production execution to Step Functions and Airflow. Ramp chose AWS-managed infrastructure, initially using Fargate before encountering startup and resource constraints, and later built a MetaflowOperator to simplify Airflow triggering and log access. After adoption, Ramp shipped eight additional models in ten months, recorded more than 6,000 Flow runs, and reports that data scientists can largely self-service while platform engineers spend less time debugging infrastructure.
Zero traffic cost for Kafka consumers
Grab's real-time data streaming platform team reduced infrastructure expenses by enabling Kafka consumers to fetch from the closest partition replica instead of partition leaders. The platform previously operated across three AWS Availability Zones with three-way replication, resulting in cross-AZ network fees that constituted half of the Kafka platform's total cost. To address this, the team performed a zero-downtime upgrade to Kafka 3.1, configured a rack-aware replica selector using AWS Availability Zone IDs, and updated consumer SDKs to populate the client rack parameter. The rollout produced a 25% drop in cross-AZ traffic within three months under steady data volumes. However, the change increased end-to-end latency by up to 500ms, created potential CPU load skews across zones, and removed graceful isolation during broker rotation maintenance.
Identifying vulnerabilities in GitHub Actions & AWS OIDC Configurations
In 2021, GitHub released OpenID Connect support for GitHub Actions to let workflows retrieve short-lived cloud session tokens instead of storing static secrets. Tinder Security Labs discovered widespread IAM role misconfigurations that allow external repositories to assume target roles and access AWS resources. The primary issue stems from IAM trust policies that authenticate the GitHub identity provider but omit strict validation on the subject claim. Researchers located exposed role identifiers via public workflow searches and confirmed that arbitrary external GitHub Actions workflows could assume them, including roles within AWS official repositories. Remediation requires configuring trust policies to restrict the audience and validate repository namespaces, while unauthorized assumptions can be detected in CloudTrail logs.
Launch Week 7 Community Highlights
Supabase reports community highlights from the past three months, covering ecosystem partnerships, self-hosting, Postgres extensions, client libraries, utilities, upstream work, and contributors. New collaborations include AWS RDS work on pg_tle and Trusted Language PL/Rust, Clickhouse bidirectional Postgres integration, LangChain vector-store and hybrid-search contributions, a Vercel and OpenAI GPT-search template, Doppler secret syncing, Redwood’s Prisma Quick Start, Dart Edge, and T3 Turbo. The update also describes self-hosting support for Logs and Edge Functions, new Digital Ocean and Stackgres entrants, and extensions including pgvector, HypoPG, pg_tle, PGroonga v3.0, LZ4 compression, and ICU support. It records new Swift, Python, and C# client libraries, cache helpers, Firebase-to-Supabase supashim, Postgres and OrioleDB patches, and recognition for contributors, while noting more than 48,000 GitHub stars.
Trusted Language Extensions for Postgres
Cloud Postgres providers often restrict extension installation until extensions are vetted, leaving some useful functionality unavailable on hosted databases. A Trusted Language Extension (TLE) is a Postgres extension written in a trusted language, allowing installation through pg_tle without placing code on the server filesystem. The source explains that conventional extensions may include SQL and functions implemented in languages such as C, while trusted languages follow PostgreSQL's security rules and cannot access raw process memory or system functions. AWS developed pg_tle, and collaboration with Supabase makes it available on Supabase and Amazon RDS for PostgreSQL 14.5 and higher; the dbdev registry can install packages such as langchain-embedding_search through SQL migrations. TLEs are presented as a way to distribute and iterate on trusted extensions without waiting for each cloud provider to preinstall them.
Graph service platform
Grab's GrabDefence team required a dedicated graph infrastructure to proactively identify mobile fraud patterns, such as multiple accounts operating on shared physical devices and suspicious financial loops. To address this, the team built a four-layer Platform as a Service that encapsulates graph database operations behind uniform RESTful APIs for OLTP search and OLAP analysis. The architecture utilizes Amazon S3 for raw data files, Amazon Neptune for graph storage, DynamoDB for schema and metadata configurations, and Kafka for streaming ingestion. When users trigger data loading tasks, the service validates entity attributes against schemas stored in DynamoDB before importing records into Neptune. This infrastructure allows investigators to traverse adjacent account IDs and visualize complex entity relationships without managing underlying database runtimes.
Automatic rule backtesting with large quantities of data
Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.
Supabase is SOC2 compliant
Supabase reports achieving SOC2 Type 1 compliance and later updates the announcement to say it is SOC2 Type 2 compliant. The post explains SOC2’s five Trust Services Criteria and distinguishes Type 1’s point-in-time audit from Type 2’s observation period, while framing customer expectations and data stewardship as the main reasons for pursuing certification. Supabase used Vanta to monitor controls and collect evidence, selected an auditor familiar with SaaS companies and Vanta, and formalized practices covering device security, access control, policies, logging, monitoring, vendor reviews, and repository protection. The process still required substantial manual evidence, and the company found that audit scope and auditor expectations mattered more than achieving a perfect automated-tool score. Supabase also launched a security center and identified HIPAA certification as a next step.
How we store and process millions of orders daily
The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.
The Serverless Server
The post explains what “serverless” means by reconstructing the machinery beneath a Lambda-like platform. It starts with bare-metal x86 servers, a Type 1 hypervisor, and services for placement, VM management, request forwarding, and function execution; dedicated customer workers, shared warm pools, rate limits, and container-style isolation address multitenancy, cold starts, and bursty demand. The runtime design uses object storage for function code, chroot, privilege dropping, namespaces, seccomp, cgroups, and freezing to constrain idle instances and workloads. Its limits on initialization, invocation duration, and idle lifecycles support migration away from unhealthy instances but disadvantage applications with expensive startup, dynamic local caches, or uncertain response times.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.