How AWS CloudFormation Turns a Template Into a Reconciled Stack
CloudFormation is a declarative reconciler: you describe the target state in a template, and it provisions, updates, and reports on the stack that results. Here is the loop.
Loading…
Topic
AWS
199 posts about AWS. Every summary links to the original.
- 199 posts
- 29 companies
- 4 explainers
See the idea
AWS explainers
Amazon EVS Architecture: Where vSAN Ends and FSx for ONTAP Begins
Amazon EVS runs VMware Cloud Foundation on AWS bare metal, so the design question becomes where each workload's data lives. This layered diagram shows how vSAN, FSx for ONTAP, and cross-Region DR divide the work.
Amazon EKS as a Platform Boundary Map
See how a shared Amazon EKS architecture separates workloads, platform services, and network connectivity so teams can scale delivery without multiplying operational ownership.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.
Gala supercharges analytics performance with ClickHouse on AWS
Blockchain gaming platform Gala struggled to ingest and query expanding volumes of telemetry and blockchain data on its Databricks-based data infrastructure, resulting in multi-minute query times and heavy engineering maintenance overhead. To solve these bottlenecks, the company migrated its data platform to ClickHouse on AWS, ingesting data through Kafka, Airbyte, Amazon S3, and Fivetran. Gala leveraged ClickHouse's indexing capabilities alongside saved queries and API endpoints to power company-wide Metabase business intelligence dashboards for non-technical teams. Following the migration completed in December 2024, data capacity expanded from 3 TB to 9 TB while query times dropped from minutes to sub-second on previously unoptimized tables. The switch also lowered initial infrastructure costs by 30 percent, with plans underway to implement ClickHouse ClickPipes for improved data processing pipelines.
Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer
Pinterest deployed a root-leaf architecture for online machine learning serving to decouple CPU-based feature preprocessing from GPU-based inference. Because the root cluster retrieved a union of features across models and fanned out the full set to every leaf, network bandwidth became a severe bottleneck. After initial tests with fbthrift LZ4 compression yielded a 20% bandwidth reduction at the cost of CPU and latency overhead, the team implemented Feature Trimmer. This mechanism uses exported model signatures as the source of truth to prune unused features on the root before transmission. By dispatching only required features per model, the system reduced network bandwidth, decreased latency, and enabled cluster rightsizing that saved over $4M annually.
Mercedes-Benz builds a cross-cloud data mesh with Delta Sharing and intelligent replication, cutting costs by 66%
Mercedes-Benz built a cross-cloud data mesh to share after-sales data between AWS and Azure while supporting its data-defined vehicle strategy. Existing direct queries incurred egress costs, and weekly full loads delayed updates for datasets that included vehicle over-the-air events and workshop visits; source tables were in Iceberg while many consumers expected Delta. Using Unity Catalog and Delta Sharing, the company federated AWS Glue tables, established cross-metastore shares, and used periodic serverless Sync Jobs with Delta Deep Clone to incrementally replicate selected tables into Azure object storage for local querying. This hybrid model preserves direct sharing for fresher workloads while reducing transfer for less time-sensitive ones. For the initial ten data products, egress costs fell 66%, update frequency improved from weekly to every second day, and the source reports an approximately 93% annual reduction in its 50-use-case comparison.
PostgresBench: A Reproducible Benchmark for Postgres Services
PostgresBench is a public, reproducible benchmarking suite created to compare transactional performance across managed PostgreSQL providers. The benchmark uses the standard pgbench tool with a TPC-B-like write-intensive workload executing across 256 clients and 16 threads for ten-minute durations. Evaluators tested five services—including Postgres managed by ClickHouse, AWS Aurora, AWS RDS, Neon, and Crunchy Bridge—under 100 GB and 500 GB scale factors using default configurations in the us-east-2 region. Across tested configurations, Postgres managed by ClickHouse delivered higher transactions per second and lower P99 latencies due to physically co-located NVMe primary storage. These results demonstrate that remote network storage layers, such as EBS volumes, introduce significant latency overhead during sustained write and fsync operations.
5 Career Myths From Women in Engineering
Three engineering leaders at Two Sigma—Angela Wang, Cecilia Ye, and Mae Santos—challenge conventional career progression models by examining five persistent industry myths. Diverse transitions across civil engineering, enterprise architecture, and customer-facing solutions architecture supplied foundational skills for modern roles in feature engineering, reliability engineering, and real-time trading optimizers. The leaders emphasize that deep domain expertise stems from direct proximity to an active system rather than cumulative years of tenure. Beyond core programming capabilities, proactive contributions such as upgrading operational standards and cultivating peer relationships drive career opportunities and leadership recognition. Adapting to emerging technologies and maintaining consistent execution allows engineers to build credibility across evolving technical environments.
Observability and OpenTelemetry: Introducing MongoDB Atlas Log Integration
During database latency spikes or unexpected outages, operations teams frequently waste valuable time downloading compressed archives and manually inspecting log files to isolate root causes. MongoDB Atlas resolves this operational friction by introducing a direct log integration feature for dedicated M10+ clusters. The system routes mongod, mongos, and audit logs directly to third-party tools such as Datadog, Splunk, Amazon S3, Google Cloud Storage, and Azure Blob Storage. Furthermore, built-in support for the OpenTelemetry standard enables teams to pipe database logs into any OTel-compatible logging backend. This setup eliminates operational silos, streamlines security auditing, and correlates database query diagnostics with application traces and infrastructure metrics in real time.
Introducing Storage Buckets on the Hugging Face Hub
Hugging Face introduces Storage Buckets as a mutable, S3-like object storage layer on the Hub for intermediate machine-learning artifacts such as checkpoints, optimizer states, processed shards, logs, and traces. Unlike versioned Models and Datasets repositories, Buckets support frequent writes, overwrites, directory synchronization, and cleanup while retaining standard Hub permissions and public or private visibility. Built on Xet, they chunk and deduplicate content across files, reducing repeated bytes, bandwidth, transfer time, and, for Enterprise customers, billed storage. The CLI and Python APIs support creation, syncing, dry runs, plans, inspection, individual file operations, and programmatic workflows; HfFileSystem and fsspec also allow compatible libraries to read and write through hf:// paths. Pre-warming can place hot data near AWS or GCP compute, while roadmap work aims to connect Buckets directly with versioned repos.
Log Drains: Now available on Pro
Supabase has launched Log Drains for its Pro tier, allowing users to send Supabase logs to their own logging backend and investigate them alongside the rest of their stack. The feature addresses the separation between Supabase logs and external observability tools by exporting events from Postgres, API Gateway, Auth, Storage, Edge Functions, and Realtime. Logs are sent over HTTP in batches of up to 250 entries or every second, whichever comes first, with Gzip compression when supported by the destination. Supported destinations include Sentry, Grafana Loki, Datadog, AWS S3, Axiom, and generic HTTP endpoints; users configure drains in the dashboard and can create multiple drains, with pricing of $60 per drain per project, $0.20 per million events, and $0.09 per GB of egress.
Minions: Stripe’s one-shot, end-to-end coding agents—Part 2
Stripe’s minions are an unattended agentic coding flow whose pull requests are human-reviewed but contain no human-written code; more than 1,300 merge weekly. The system runs agents in AWS EC2-based devboxes that are prewarmed to become usable within about 10 seconds, providing parallel, predictable, and isolated environments. Its custom harness uses code-defined blueprints that combine deterministic nodes, such as linting and pushing changes, with agent loops for tasks such as implementation and fixing CI failures. Minions receive automated feedback through local linting, one full CI iteration, and a possible second repair attempt, while devbox isolation and security controls limit destructive actions; Stripe presents the approach as an extension of long-running investments in human developer productivity.
Supabase incident on February 12, 2026
Supabase reports a 3-hour-and-42-minute outage on February 12–13, 2026, affecting every service in its us-east-2 region, including Postgres, Auth, Data APIs, Edge Functions, Storage, and Realtime. A deployment of an internal monitoring service enabled AWS VPC Block Public Access in block-bidirectional mode, blocking internet gateway traffic across production VPCs in the region. The incident response was delayed because alerts pointed toward shared services, the relevant CloudTrail event was inconspicuous, pre-production lacked us-east-2, and infrastructure specialists were not involved initially. Supabase restored connectivity by rolling back and removing the monitoring stack, then audited regions, added AWS Organizations policies and IaC safeguards, and planned account isolation, external connectivity probes, production-parity testing, and improved escalation.
Balancing cost and reliability for Spark on Kubernetes
Notion describes how it reduced Spark compute costs on Kubernetes while maintaining reliability across production workloads. After moving from fixed EC2 clusters on Amazon EMR to EMR on EKS, it used Karpenter, EKS Auto Mode, and the MostAllocated scheduler for dynamic provisioning and bin-packing. Spot Instances then exposed a failure mode: one interruption could remove many bin-packed executors, causing shuffle loss and expensive recomputation. Spot Balancer addresses this by labeling executor creation through a Kubernetes webhook and maintaining each job’s configured spot-to-on-demand ratio, with per-job counts stored in Redis. Named stability weights make the cost-reliability trade-off explicit, and reported savings reached 60–90 percent while jobs that previously failed from interruptions finished reliably.
Docker lazy loading at Grab: Accelerating container startup times
Large Docker images for Airflow and Spark Connect caused minute-scale downloads, delaying cold starts and weakening autoscaling responsiveness. The post describes testing eStargz and Seekable OCI (SOCI) with containerd remote snapshotters, which expose registry-backed layers through a virtual lowerdir and fetch file content on demand. In production, SOCI delivered 30–40% faster startup times for Airflow and Spark Connect, while benchmark results showed SOCI matching standard images’ 5.0-second Airflow application startup; eStargz took 25.0 seconds. Configuration tuning increased download concurrency, unpack concurrency, and chunk size, reducing fresh-node image download time from 60 seconds to 24, a 60% improvement. Grab chose SOCI because Bottlerocket supports it natively, while noting that lazy loading redistributes rather than removes download work and requires workload-specific tuning.
PaaS + IaaS: Heroku and AWS for Cloud-Native Enterprise Agility
Modern enterprises can combine AWS IaaS infrastructure scale with Heroku’s managed PaaS to reduce operational work and developer cognitive load while retaining control for specialized workloads. Heroku supports Git-based deployment, managed services including Apache Kafka and Heroku Postgres, polyglot applications, and isolated environments such as Private Spaces and Shield Private Spaces. In Salesforce’s example, five applications serving more than 76,000 employees used Heroku to cut expense approval time from two days to two hours and recover $2 million in unused licenses. The architecture also combines Heroku AppLink, Heroku Connect, Heroku Fir’s OpenTelemetry support, and managed Dynos, while AWS remains the stated choice for highly specialized AI training requiring raw GPU access.
Celebrating Excellence: MongoDB Global Partner Awards 2025
MongoDB announced the recipients of its 2025 Global Partner Awards, recognizing cloud providers, systems integrators, and technology vendors for driving enterprise modernization and artificial intelligence adoption. Microsoft earned Global Cloud Partner of the Year for joint integrations linking MongoDB Atlas on Azure with native Microsoft services across healthcare, telecommunications, and financial services. Amazon Web Services received the Global AI Cloud Partner award, highlighted by a joint deployment with Novo Nordisk that reduced a key workflow from twelve weeks to ten minutes using Amazon Bedrock and Atlas. Confluent was named Global Tech Partner of the Year with over 550 joint customer deployments focused on event-driven streaming and multi-agent systems alongside LangChain. Additional honorees included Google Cloud, Accenture, BigID, Pureinsights, gravity9, IBM, and Alibaba Cloud for their contributions across public sector solutions, database-as-a-service offerings, and generative AI frameworks.
庆祝卓越：MongoDB 全球合作伙伴奖 2025
MongoDB announced its 2025 Global Partner Award recipients to recognize organizations driving AI adoption, legacy system modernization, and collaborative market expansion. Microsoft earned Global Cloud Partner for Azure integrations, while Amazon Web Services took Global AI Cloud Partner after assisting Novo Nordisk in cutting a key workflow from twelve weeks to ten minutes with Amazon Bedrock. Google Cloud received the Global Cloud GTM Partner award, and Confluent was named Global Technology Partner with over 550 joint deployments. LangChain and Pureinsights received honors for their integrations supporting retrieval-augmented generation and search solutions. Additional awards celebrated Accenture, BigID, gravity9, IBM, and Alibaba Cloud for their enterprise and public-sector impact.
우수성을 기념하기: 2025년 MongoDB 글로벌 파트너 어워드
MongoDB announced the recipients of its 2025 Global Partner Awards to recognize partner contributions across cloud modernization, artificial intelligence, and enterprise integration. Microsoft received the Global Cloud Partner award for Azure integrations, while Amazon Web Services secured the Global AI Cloud Partner award following joint generative AI implementations such as reducing workflow durations for Novo Nordisk. Google Cloud earned the Global Cloud GTM Partner distinction through shared sales development programs, and Accenture took Global SI Partner honors after establishing a dedicated engineering Center of Excellence. Confluent was named Global Tech Partner with more than 550 joint streaming deployments, and LangChain was recognized as Global AI Tech Partner for vector search and agentic frameworks. The awards demonstrate partner-led technical solutions spanning distributed databases, data streaming pipelines, and scalable enterprise cloud architectures.
Celebrando la Excelencia: Premios Globales de Emparejar de MongoDB 2025
MongoDB presented its 2025 Global Partner Awards to honor partner organizations across cloud infrastructure, systems integration, generative AI, and enterprise data management. Microsoft received the Global Cloud Partner award for joint go-to-market solutions integrating MongoDB Atlas with native Azure services across healthcare, telecommunications, and financial services. Amazon Web Services earned the Global AI Cloud Partner award after delivering an Amazon Bedrock and MongoDB Atlas workflow implementation for Novo Nordisk that reduced processing time from twelve weeks to ten minutes. Confluent was recognized as Global Technology Partner for exceeding 550 joint deployments leveraging Apache Kafka for real-time data streaming and event-driven AI architectures. Other honored partners include Google Cloud for joint sales development programs, Accenture for software engineering centers of excellence, LangChain for retrieval-augmented generation tooling, and IBM for Watsonx.ai integrations.
Hommage à l’excellence : MongoDB Global Partner Awards 2025
MongoDB announced the recipients of its 2025 Global Partner Awards, recognizing enterprise collaboration across cloud infrastructure, artificial intelligence, and systems modernization. Microsoft received the Global Cloud Partner award for joint integrations with Azure, while Amazon Web Services earned the Global AI Cloud Partner award following deployments combining Amazon Bedrock and MongoDB Atlas. Google Cloud was recognized for joint go-to-market initiatives, and Accenture received honours for establishing a dedicated MongoDB Center of Excellence within its software engineering division. Additional recognitions included Confluent for event-driven streaming deployments exceeding 550 customer implementations, BigID for data governance, gravity9 for modernization services, and Alibaba Cloud for managed database services. IBM and Accenture Federal Services were also acknowledged for strategic enterprise impact across mainframe architectures and public sector programs.
Seamlessly connect Stripe events to your frontend
Stripe events can connect backend state changes to responsive frontend interfaces for payments, subscriptions, connected accounts, and Stripe Terminal workflows. The post contrasts polling, which can increase traffic, latency, and delta-computation complexity, with push subscriptions, while noting that a hybrid of initial API loading and incremental updates is common. It describes webhook endpoints and Amazon EventBridge as ingestion choices, then emphasizes idempotency, ordering, duplicate detection, and failure handling. Frontend delivery options include WebSockets, MQTT, gRPC, Server-Sent Events, polling, AWS AppSync, and Momento, selected according to messaging pattern, direction, latency, scale, and operational trade-offs. An abbreviated lounge example uses a PaymentIntent and Terminal reader, routes payment_intent.requires_payment_method through EventBridge to AWS AppSync Events, and lets a React client mark the reader ready to collect payment.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.