How AWS CloudFormation Turns a Template Into a Reconciled Stack
CloudFormation is a declarative reconciler: you describe the target state in a template, and it provisions, updates, and reports on the stack that results. Here is the loop.
Loading…
Topic
AWS
201 posts about AWS. Every summary links to the original.
- 201 posts
- 29 companies
- 4 explainers
See the idea
AWS explainers
Amazon EVS Architecture: Where vSAN Ends and FSx for ONTAP Begins
Amazon EVS runs VMware Cloud Foundation on AWS bare metal, so the design question becomes where each workload's data lives. This layered diagram shows how vSAN, FSx for ONTAP, and cross-Region DR divide the work.
Amazon EKS as a Platform Boundary Map
See how a shared Amazon EKS architecture separates workloads, platform services, and network connectivity so teams can scale delivery without multiplying operational ownership.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How Vercel used Cursor to build Queues
Vercel built its durable event streaming system, Vercel Queues, entirely with Cursor, using agents across three rewrites and validating changes against real AWS environments. Engineers had agents work on the Queues API, connect to AWS, inspect DynamoDB and Kinesis side effects, inject faults, simulate bugs, and verify results, while Cursor retained context through long sessions and compaction events. The workflow also supported model switching, parallel planning agents, inline review, stacked PRs, and asynchronous Cloud Agents. Among Vercel’s Cursor power users, PR throughput increased 54%, PR cycle time decreased 89%, resources per feature fell 27%, and the company reports 104 hours saved per developer annually. Product managers also use Cursor for repository analysis, product analytics, and small fixes without drawing on engineering resources.
amazon.science
Amazon is investing in the Lean Focused Research Organization
As AI agents handle high-stakes operational and financial decisions, traditional software testing fails to provide exhaustive correctness guarantees across all possible inputs. Mathematical proof offers deterministic verification that systems cannot behave incorrectly, but requires accessible tooling to function at modern software scale. To address this, Amazon announced substantial long-term financial backing for the Lean Focused Research Organization, representing the largest donation in the organization's history. Internally, Amazon relies on Lean for verifying differential-privacy mechanisms in AWS Clean Rooms, hardware compilation in AWS Neuron, and policy constraints within Amazon Bedrock AgentCore. Developing Lean externally through an open-source research organization provides independent auditability for regulators and accelerates community-driven tooling for neurosymbolic AI safety.
Connect Amazon S3 data to Databricks with Delegated IAM Permissions
Databricks introduces a simpler flow for connecting Amazon S3 to Unity Catalog through an external location, addressing the manual work previously required to establish governed read and write access. Instead of writing 140-line IAM trust policies, configuring bucket permissions, deploying CloudFormation templates, and switching between consoles, users specify a bucket and access level, then verify permissions in AWS. AWS IAM temporary delegation lets Databricks provision a least-privilege IAM role, configure its cross-account trust policy, register the external location, and enable Auto Loader and File Events. The delegated authorization is time-bounded and expires after setup; provisioning actions are logged in AWS CloudTrail, and users without sufficient permissions can request access from an AWS administrator in the flow. The result is a single-session setup intended to reduce configuration errors while supporting governed S3 access for ingestion, pipelines, analytics, and LTAP workloads.
Building a serverless AI assistant at Pelago: concept to care in two weeks
Pelago engineered a serverless, event-driven AI assistant on AWS within two weeks to provide substance use recovery coaches with contextual response suggestions. Operating under strict Protected Health Information constraints, the system maintains clinical safety through human-in-the-loop review and keeps data confined to an Amazon VPC. Incoming messages flow via AWS AppSync and Amazon DynamoDB to an Amazon SNS topic, which fans out to independent AWS Lambda functions. The asynchronous Chat Assistant Lambda queries history, invokes Amazon Bedrock, and persists suggestions into Amazon RDS MySQL in under 10 seconds. When coaches access a conversation via Amazon API Gateway, pre-generated suggestions load in under 100 milliseconds.
Securing Infrastructure at Scale: Introducing Pinterest’s Resource Provisioner Pipeline (RPP)
Managing Infrastructure as Code across distributed repositories presents significant security and operational challenges. Pinterest developed the Resource Provisioner Pipeline (RPP), a proprietary Terraform execution engine, to securely deploy and manage AWS infrastructure across hundreds of workspaces. The pipeline utilizes centralized GitHub composite actions triggered by pull request events and enforces dual human approvals. To maintain least privilege, RPP uses GitHub OpenID Connect authentication to assume a central identity before validating repository paths and down-scoping to specific team execution roles. Automated checks include backend state validation, linting, plan generation, Semgrep static analysis, and optional LocalStack testing before deliberate comment-driven deployments execute.
Building multi-Region resiliency for AWS CloudFormation custom resource deployment
AWS CloudFormation custom resources extend template provisioning capabilities but lack built-in multi-Region coordination, risking duplicate executions and unhandled failures during regional outages. To solve this limitation, an active-active architecture leverages Amazon Simple Notification Service topics to fan out lifecycle events across primary and secondary infrastructure Regions. The primary Region processes events immediately, whereas the secondary Region processes incoming messages after a deliberate delay implemented via Amazon Simple Queue Service. Coordination relies on Amazon DynamoDB Global Tables with conditional writes to manage distributed locking and enforce idempotency. When failures occur in the primary Region, Amazon CloudWatch alarms signal Amazon Application Recovery Controller to automate failover without manual intervention.
Architecting offline-first generative AI applications for edge deployments using AWS services
Industrial edge environments often face significant unplanned downtime yet lack reliable cloud connectivity to run large-scale artificial intelligence models. To address this limitation, an offline-first reference architecture shifts generative inference to edge devices while leveraging cloud services for preparation and lifecycle management. The workflow uses Amazon Bedrock with Amazon Nova Pro to create labeled dataset pairs, Amazon SageMaker AI Pipelines to fine-tune a compact model, and AWS IoT Greengrass to orchestrate edge deployments. Local execution relies on Ollama, Strands Agents, and a CPU-based ChromaDB retrieval pipeline that consumes zero GPU VRAM. In evaluations across three foundation model judges, the fine-tuned gpt-oss-20b model combined with retrieval-augmented generation consistently outperformed the base model configuration across accuracy, completeness, and relevance metrics.
Automate custom PII detection at scale with Amazon Macie and Step Functions
Regulated organizations ingesting large data volumes must detect and classify standard and domain-specific personally identifiable information to satisfy compliance mandates like GDPR, HIPAA, CCPA, and PCI DSS. To eliminate manual inspection, an event-driven architecture orchestrates Amazon Macie scans for files landing in Amazon S3. Amazon EventBridge captures upload events and invokes AWS Step Functions, which manages Lambda functions to stage objects, trigger Macie classification jobs with custom regex identifiers, and poll for results. The pipeline separates data across a three-bucket pattern representing raw, staged, and scanned states to prevent mixing unprocessed data with validated assets. Finally, the workflow exports timestamped compliance reports in CSV and JSON formats while alerting security teams via Amazon SNS on high-severity findings.
Eclipse Dataspace Components on AWS: Cost optimization strategies
Deploying Eclipse Dataspace Components (EDC) connectors on AWS presents challenges in forecasting infrastructure expenses without clear benchmarks. Estimated costs differ substantially between business-critical setups and non-critical environments based on compute and database sizing. A business-critical configuration relying on Amazon Aurora PostgreSQL db.r6g.large and standard Amazon ECS with AWS Fargate totals an estimated 387.00 USD monthly in the eu-south-2 region. Sizing down to db.t4g.medium and switching to AWS Fargate Spot lowers the non-critical monthly expense to 164.00 USD, representing a 58% reduction. Core cost drivers remain baseline infrastructure components, while storage, API calls, and data transfers contribute marginally under baseline assumptions.
Why Trainy migrated from Amazon RDS Postgres to ClickHouse Managed Postgres
Trainy’s Pluto is an open-source experiment tracker that separates analytics metrics in ClickHouse Cloud from relational run metadata in Postgres. Its original metadata store, Amazon RDS Postgres, was too expensive for the volume observed and contributed to slower initial page loads, so Trainy migrated roughly 40–50 GB to ClickHouse Managed Postgres, which uses local NVMe storage and keeps both databases in one account. After migration, initial page loads became 2–3 times faster, while a complex query reported as taking 10–15 minutes previously returned in five seconds; visualization queries over runs with hundreds of thousands of steps typically return in 50–100 milliseconds at the P50. Trainy reports 100% reliable ingestion with zero dropped data points and says the unified stack could support future joins between GPU metrics and run metadata, as well as governed agent access.
How Bullet uses ClickHouse Cloud to give DeFi's fastest exchange real-time analytics
Bullet operates a DeFi exchange for perpetuals, spot trading, and lending, with trades settled on a blockchain while users retain custody of their assets. Its sub-millisecond trading system produces more than 150 million event rows per hour, creating a need for historical indexing, real-time analytics, and operational monitoring beyond the blockchain’s state-machine model. ClickHouse Cloud consumes events from Kafka, stores them in dedicated tables, and supports user features, leaderboard and reward calculations, P&L charts, audit logs, and Grafana-backed alerts. Replacing Databricks reduced query latency from 10–15 seconds to milliseconds, improved data freshness from 1–2 hours to under 5 seconds, and removed a separate DynamoDB serving layer at comparable cost while handling 1,000 times more data across staging, testnet, and mainnet.
Shipyard: How We Built Slack’s Next-Generation EC2 Platform
Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows.
How Insider One uses ClickHouse Cloud to power real-time customer engagement at scale
Insider One’s Unified Customer Database (UCD) unifies event data, user attributes, identity resolution, and profiles across channels to support real-time segmentation, personalization, and analytics. The platform initially ran ClickHouse in a self-managed AWS deployment, but growth brought larger datasets, more concurrent queries, and increasing work tuning clusters and balancing performance against cost. In a proof of concept, the team migrated an anonymized dataset from one of its largest customers to ClickHouse Cloud and replayed UI segmentation, audience-count, and scheduled-campaign workloads. Average query latency fell 50–60% without significant changes, reaching 70–75% for the largest customers, while latency percentiles became more predictable; the managed service also reduced operational burden and gave the team more capacity for product development and its AI-oriented roadmap.
Introducing Atlas Gen2 on AWS and GCP for M30+ Dedicated Clusters
MongoDB Atlas introduced Gen2 for M30+ Dedicated Clusters on AWS and GCP, addressing the common inefficiency of overprovisioning storage to meet IOPS requirements. Gen2 runs on newer ARM-based processors, delivering a statistically significant 15-20% reduction in CPU utilization and approximately 45% lower read and write latency in internal pilots. It introduces extended standard IOPS that can be provisioned independently from storage capacity, allowing teams to scale performance without adding unused storage and gain more precise control over infrastructure costs. No application changes or migration are required; teams retain the same Atlas API, connection strings, and operational model while benefiting from newer infrastructure underneath. Gen2 is available in supported AWS and GCP regions across APAC and Europe, with limited U.S. availability on GCP, and existing customers can evaluate upgrade opportunities for eligible M30+ clusters.
How Artemis Security runs 69x faster detection queries with ClickHouse Cloud
Artemis Security uses ClickHouse Cloud on AWS for real-time threat detection across many terabytes of compressed logs, tens of billions of daily rows, and queries over trillions of rows. To reduce repeated scans, its engineers coalesce hundreds of detection-rule predicates into one query, tagging matching rows by rule ID; batches of more than 100 rules now run in under a second, with a measured 69x speedup. A Claude Code skill called /poke connects ClickHouse system tables, CloudWatch, Datadog, application data, and source code to investigate infrastructure issues through natural-language prompts. For investigative queries over semi-structured JSON, Artemis extracts frequently queried fields into a parallel materialized table and uses a hybrid two-step lookup, producing 30–60x faster queries and up to 400x lower CPU consumption.
Clever ingests 200x more logs at the same cost with ClickHouse Cloud
Clever, an identity platform for K-12 education, faced a fragmented logging setup as its 400 services and 200 AWS Lambdas generated about 150 TB of uncompressed logs monthly. Datadog indexed only about 10% and retained it for three days, while the remainder was in Athena and required SQL. Clever migrated to ClickHouse Cloud on AWS, indexing all logs with 60-day retention and reporting 200× more searchable log volume at no additional cost, with 10× compression. Its design extracts frequently queried JSON fields into columns, stores remaining attributes in a map alongside raw JSON, uses skipping indexes, date partitions, a 60-day TTL, and reverse Unix timestamp ordering; this ordering produced an 8–10× query speedup. A custom query layer translates LogQL to SQL, while separate development and production services and multi-region deployment address isolation and availability.
ClickHouse achieves AWS Cloud Operations Competency
ClickHouse has achieved the AWS Cloud Operations Competency in the Monitoring and Observability category, recognizing validated expertise in operating reliable AWS systems with logs, metrics, and traces. The post describes observability as a high-volume, high-cardinality, high-concurrency workload where legacy systems may require sampling, field reduction, or short retention. ClickHouse Cloud uses columnar storage, compression, parallel queries, compute-compute separation, and ClickPipes integrations with Amazon Kinesis, Amazon MSK, and Amazon S3 to support full-fidelity telemetry. Customer examples include Modal’s 1 to 2 million events per minute, Exabeam’s 1.2 million events per second per region, and Qonto’s reduction of 231 TB to 376 GB. The company also describes MCP servers, AI Notebooks, and managed ClickStack as investments for collaborative incident investigation.
10 Years of MongoDB Atlas: Built for What’s Next
Launched in 2016, MongoDB Atlas was designed to bring MongoDB's developer experience to the cloud while eliminating operational overhead for production environments. Over ten years, Atlas expanded beyond a managed database into a multi-cloud data platform that supports multi-document ACID transactions, Queryable Encryption, native search, and stream processing. The platform introduced Atlas Vector Search in 2023 and acquired Voyage AI in 2025 to embed advanced embedding and reranking models natively. These integrated capabilities address the requirements of retrieval-augmented generation and autonomous AI agents without relying on disparate vector stores or sync pipelines. Atlas now serves more than three trillion queries daily across over 125 AWS, Google Cloud, and Microsoft Azure regions for nearly 70,000 customers.
Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models
Pinterest’s foundation ranking models store approximately 99% of their parameters in embedding tables, causing cross-node NCCL all-to-all communication to severely bottleneck multi-node distributed training. Initial multi-node scaling stalled at 1.21x on four nodes even after adopting AWS Elastic Fabric Adapter for OS-bypass networking. To resolve this, engineers profiled GPU activity with PyTorch Profiler and systematically addressed communication bottlenecks across multiple layers. The team implemented FP8 quantized communications using FBGEMM, balanced table sharding across GPUs, halved embedding dimensions while doubling row counts, and adopted a 2D parallel communication topology alongside PyTorch Distributed Checkpoint. These combined interventions achieved near-linear scaling factors of 3.9x on four nodes and 7.5x on eight nodes across 64 GPUs.
amazon.science
The fuel of the future is already here: Why TRISO matters
Amazon is investing in next-generation nuclear technology powered by tristructural isotropic (TRISO) fuel particles to satisfy the increasing energy demands of cloud computing and artificial intelligence infrastructure. Each sub-millimeter TRISO particle encloses a high-assay low-enriched uranium kernel within a porous carbon buffer and a three-layer ceramic shell featuring silicon carbide that acts as a robust miniature containment vessel. Testing by the U.S. Department of Energy demonstrated that irradiated particles withstand temperatures of 1600°C for 300 hours without detectable failures. In pebble-bed designs like the X-energy Xe-100 reactor, graphite spheres containing TRISO fuel circulate continuously through the core to enable online refueling, passive heat removal, and modular 80-megawatt deployments at facilities like the Cascade Energy Center.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.