---
title: "AWS"
description: "205 posts about AWS, summarised, each linking to the original."
---

# AWS
> 205 posts about AWS, summarised, each linking to the original.

## Articles

### [What is WAL backpressure, and why does ClickHouse Managed Postgres need it?](https://yomu.fyi/post/what-is-wal-backpressure-and-why-does-clickhouse-managed-postgres-need.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 5, 2026
- Type: Explainer

Postgres writes every change to the write-ahead log (WAL), and Managed Postgres uploads completed segments to object storage for point-in-time recovery; if uploads lag, retained WAL can fill the disk and cause a PANIC shutdown. ClickHouse Managed Postgres counters this with a data-plane systemd timer that checks pending segments every 15 seconds and uses cgroup v2 I/O controls to throttle client write bandwidth at 80%, 50%, or 20% of provisioned throughput as backlog thresholds rise. The drain path—including the archiver, checkpointer, and related processes—is placed in an immune cgroup, while client backends are throttled and reads remain uncapped. In a test with a 500 MB/s gp3 volume and an archive limited to 4 MB/s, the backlog reached 3,433 segments and 53 GiB before recovery. Once the limit was removed, the archiver cleared the queue, the throttle released automatically, and the disk stayed below 33% full.


### [34 Amazon Research Awards Build on Trainium recipients announced](https://yomu.fyi/post/34-amazon-research-awards-build-on-trainium-recipients-announced.md)
- Company: amazon.science
- Author: Amazon Research Awards team
- Published: Aug 5, 2026
- Type: Announcement

Amazon announced 34 recipients from 30 universities for the Fall 2025 Build on Trainium: Responsible AI call for proposals. Part of a 110 million dollar credit initiative, the program funds academic research across new model architectures, machine learning libraries, optimizations, and distributed systems running on AWS Trainium hardware. The funded projects center on five key areas: AI safety and alignment, multilingual language models, representation engineering, sustainability with small language models, and deep learning for synthetic data generation. Awardees receive AWS Promotional Credits, access to more than 700 Amazon public datasets, assigned Amazon research contacts, and dedicated technical training resources. Highlighted efforts include parallelization strategies for trillion-parameter mixture-of-experts models scaling across 1,024 Trainium chips and inference optimization frameworks designed to enhance token efficiency.


### [How Mercado Libre rebuilt its observability platform on ClickHouse Cloud with 50x faster trace queries](https://yomu.fyi/post/how-mercado-libre-rebuilt-its-observability-platform-on-clickhouse-clo.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 4, 2026
- Type: Problem & solution

Mercado Libre rebuilt its O11y events observability platform to answer granular business-level questions such as why a payment failed across a large microservice environment. The previous storage-layer design performed transformation, filtering, and aggregation at query time, causing queries to run longer than five minutes and making high-cardinality analysis expensive. The team moved storage to ClickHouse Cloud on AWS while retaining its existing ingestion path, and shifted heavy work into the database with materialized views, layered retention, and a dedicated lookup table for high-cardinality filtering. O11y events uses OpenTelemetry instrumentation with business-flow attributes and samples all traffic for critical applications. The resulting system reduced trace-query time to about four seconds, a 50x speedup, achieved up to 89% compression, and scaled from 7 million to 400 million spans per minute in ClickHouse.


### [Removing the Security Barrier to Agentforce Adoption](https://yomu.fyi/post/removing-the-security-barrier-to-agentforce-adoption.md)
- Company: [Salesforce](https://yomu.fyi/company/salesforce.md)
- Author: Scott Nyberg
- Published: Aug 3, 2026
- Type: Problem & solution

Enterprise AI adoption across regulated industries stalls when internal governance mandates private, dedicated network paths for sensitive data. To eliminate the need for manual VPN tunnels and complex proxy layers, Salesforce re-engineered Private Connect into a multi-cloud networking platform connecting customer environments to Hyperforce services. The Private Connect v2.0 architecture utilizes private links, transit gateways, and direct endpoint routing alongside an automated control plane for rapid Hyperforce regional rollouts. In addition to supporting cross-substrate interconnects for Azure and decoupling endpoint management, the platform incorporates a generic connector framework supporting tools like Snowflake, Databricks, and Kafka. Today, the system provisions secure connectivity in under 30 minutes while processing roughly 120 TB of data and 683 million requests monthly across 15 AWS regions.


### [How Vercel used Cursor to build Queues](https://yomu.fyi/post/how-vercel-used-cursor-to-build-queues.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Jul 28, 2026
- Type: Explainer

Vercel built its durable event streaming system, Vercel Queues, entirely with Cursor, using agents across three rewrites and validating changes against real AWS environments. Engineers had agents work on the Queues API, connect to AWS, inspect DynamoDB and Kinesis side effects, inject faults, simulate bugs, and verify results, while Cursor retained context through long sessions and compaction events. The workflow also supported model switching, parallel planning agents, inline review, stacked PRs, and asynchronous Cloud Agents. Among Vercel’s Cursor power users, PR throughput increased 54%, PR cycle time decreased 89%, resources per feature fell 27%, and the company reports 104 hours saved per developer annually. Product managers also use Cursor for repository analysis, product analytics, and small fixes without drawing on engineering resources.


### [Amazon is investing in the Lean Focused Research Organization](https://yomu.fyi/post/amazon-is-investing-in-the-lean-focused-research-organization.md)
- Company: amazon.science
- Author: Byron Cook, Shawn Bice
- Published: Jul 26, 2026
- Type: Announcement

As AI agents handle high-stakes operational and financial decisions, traditional software testing fails to provide exhaustive correctness guarantees across all possible inputs. Mathematical proof offers deterministic verification that systems cannot behave incorrectly, but requires accessible tooling to function at modern software scale. To address this, Amazon announced substantial long-term financial backing for the Lean Focused Research Organization, representing the largest donation in the organization's history. Internally, Amazon relies on Lean for verifying differential-privacy mechanisms in AWS Clean Rooms, hardware compilation in AWS Neuron, and policy constraints within Amazon Bedrock AgentCore. Developing Lean externally through an open-source research organization provides independent auditability for regulators and accelerates community-driven tooling for neurosymbolic AI safety.


### [Connect Amazon S3 data to Databricks with Delegated IAM Permissions](https://yomu.fyi/post/connect-amazon-s3-data-to-databricks-with-delegated-iam-permissions.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Gordon Wang
- Published: Jul 23, 2026
- Type: Announcement

Databricks introduces a simpler flow for connecting Amazon S3 to Unity Catalog through an external location, addressing the manual work previously required to establish governed read and write access. Instead of writing 140-line IAM trust policies, configuring bucket permissions, deploying CloudFormation templates, and switching between consoles, users specify a bucket and access level, then verify permissions in AWS. AWS IAM temporary delegation lets Databricks provision a least-privilege IAM role, configure its cross-account trust policy, register the external location, and enable Auto Loader and File Events. The delegated authorization is time-bounded and expires after setup; provisioning actions are logged in AWS CloudTrail, and users without sufficient permissions can request access from an AWS administrator in the flow. The result is a single-session setup intended to reduce configuration errors while supporting governed S3 access for ingestion, pipelines, analytics, and LTAP workloads.


### [Building a serverless AI assistant at Pelago: concept to care in two weeks](https://yomu.fyi/post/building-a-serverless-ai-assistant-at-pelago-concept-to-care-in-two-we.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Anton Aleksandrov
- Published: Jul 22, 2026
- Type: Problem & solution

Pelago engineered a serverless, event-driven AI assistant on AWS within two weeks to provide substance use recovery coaches with contextual response suggestions. Operating under strict Protected Health Information constraints, the system maintains clinical safety through human-in-the-loop review and keeps data confined to an Amazon VPC. Incoming messages flow via AWS AppSync and Amazon DynamoDB to an Amazon SNS topic, which fans out to independent AWS Lambda functions. The asynchronous Chat Assistant Lambda queries history, invokes Amazon Bedrock, and persists suggestions into Amazon RDS MySQL in under 10 seconds. When coaches access a conversation via Amazon API Gateway, pre-generated suggestions load in under 100 milliseconds.


### [Securing Infrastructure at Scale: Introducing Pinterest’s Resource Provisioner Pipeline (RPP)](https://yomu.fyi/post/securing-infrastructure-at-scale-introducing-pinterest-s-resource-prov.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: Jul 22, 2026
- Type: Problem & solution

Managing Infrastructure as Code across distributed repositories presents significant security and operational challenges. Pinterest developed the Resource Provisioner Pipeline (RPP), a proprietary Terraform execution engine, to securely deploy and manage AWS infrastructure across hundreds of workspaces. The pipeline utilizes centralized GitHub composite actions triggered by pull request events and enforces dual human approvals. To maintain least privilege, RPP uses GitHub OpenID Connect authentication to assume a central identity before validating repository paths and down-scoping to specific team execution roles. Automated checks include backend state validation, linting, plan generation, Semgrep static analysis, and optional LocalStack testing before deliberate comment-driven deployments execute.


### [Building multi-Region resiliency for AWS CloudFormation custom resource deployment](https://yomu.fyi/post/building-multi-region-resiliency-for-aws-cloudformation-custom-resourc.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Raman Pujani
- Published: Jul 22, 2026
- Type: Problem & solution

AWS CloudFormation custom resources extend template provisioning capabilities but lack built-in multi-Region coordination, risking duplicate executions and unhandled failures during regional outages. To solve this limitation, an active-active architecture leverages Amazon Simple Notification Service topics to fan out lifecycle events across primary and secondary infrastructure Regions. The primary Region processes events immediately, whereas the secondary Region processes incoming messages after a deliberate delay implemented via Amazon Simple Queue Service. Coordination relies on Amazon DynamoDB Global Tables with conditional writes to manage distributed locking and enforce idempotency. When failures occur in the primary Region, Amazon CloudWatch alarms signal Amazon Application Recovery Controller to automate failover without manual intervention.


### [Architecting offline-first generative AI applications for edge deployments using AWS services](https://yomu.fyi/post/architecting-offline-first-generative-ai-applications-for-edge-deploym.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Ahmed Ewis
- Published: Jul 22, 2026
- Type: Problem & solution

Industrial edge environments often face significant unplanned downtime yet lack reliable cloud connectivity to run large-scale artificial intelligence models. To address this limitation, an offline-first reference architecture shifts generative inference to edge devices while leveraging cloud services for preparation and lifecycle management. The workflow uses Amazon Bedrock with Amazon Nova Pro to create labeled dataset pairs, Amazon SageMaker AI Pipelines to fine-tune a compact model, and AWS IoT Greengrass to orchestrate edge deployments. Local execution relies on Ollama, Strands Agents, and a CPU-based ChromaDB retrieval pipeline that consumes zero GPU VRAM. In evaluations across three foundation model judges, the fine-tuned gpt-oss-20b model combined with retrieval-augmented generation consistently outperformed the base model configuration across accuracy, completeness, and relevance metrics.


### [Automate custom PII detection at scale with Amazon Macie and Step Functions](https://yomu.fyi/post/automate-custom-pii-detection-at-scale-with-amazon-macie-and-step-func.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Aishwariya Khiani
- Published: Jul 22, 2026
- Type: Tutorial

Regulated organizations ingesting large data volumes must detect and classify standard and domain-specific personally identifiable information to satisfy compliance mandates like GDPR, HIPAA, CCPA, and PCI DSS. To eliminate manual inspection, an event-driven architecture orchestrates Amazon Macie scans for files landing in Amazon S3. Amazon EventBridge captures upload events and invokes AWS Step Functions, which manages Lambda functions to stage objects, trigger Macie classification jobs with custom regex identifiers, and poll for results. The pipeline separates data across a three-bucket pattern representing raw, staged, and scanned states to prevent mixing unprocessed data with validated assets. Finally, the workflow exports timestamped compliance reports in CSV and JSON formats while alerting security teams via Amazon SNS on high-severity findings.


### [Eclipse Dataspace Components on AWS: Cost optimization strategies](https://yomu.fyi/post/eclipse-dataspace-components-on-aws-cost-optimization-strategies.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Jorge Hernández Suárez
- Published: Jul 17, 2026
- Type: Explainer

Deploying Eclipse Dataspace Components (EDC) connectors on AWS presents challenges in forecasting infrastructure expenses without clear benchmarks. Estimated costs differ substantially between business-critical setups and non-critical environments based on compute and database sizing. A business-critical configuration relying on Amazon Aurora PostgreSQL db.r6g.large and standard Amazon ECS with AWS Fargate totals an estimated 387.00 USD monthly in the eu-south-2 region. Sizing down to db.t4g.medium and switching to AWS Fargate Spot lowers the non-critical monthly expense to 164.00 USD, representing a 58% reduction. Core cost drivers remain baseline infrastructure components, while storage, API calls, and data transfers contribute marginally under baseline assumptions.


### [Why Trainy migrated from Amazon RDS Postgres to ClickHouse Managed Postgres](https://yomu.fyi/post/why-trainy-migrated-from-amazon-rds-postgres-to-clickhouse-managed-pos.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jul 15, 2026
- Type: Problem & solution

Trainy’s Pluto is an open-source experiment tracker that separates analytics metrics in ClickHouse Cloud from relational run metadata in Postgres. Its original metadata store, Amazon RDS Postgres, was too expensive for the volume observed and contributed to slower initial page loads, so Trainy migrated roughly 40–50 GB to ClickHouse Managed Postgres, which uses local NVMe storage and keeps both databases in one account. After migration, initial page loads became 2–3 times faster, while a complex query reported as taking 10–15 minutes previously returned in five seconds; visualization queries over runs with hundreds of thousands of steps typically return in 50–100 milliseconds at the P50. Trainy reports 100% reliable ingestion with zero dropped data points and says the unified stack could support future joins between GPU metrics and run metadata, as well as governed agent access.


### [How Bullet uses ClickHouse Cloud to give DeFi's fastest exchange real-time analytics](https://yomu.fyi/post/how-bullet-uses-clickhouse-cloud-to-give-defi-s-fastest-exchange-real.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jul 15, 2026
- Type: Problem & solution

Bullet operates a DeFi exchange for perpetuals, spot trading, and lending, with trades settled on a blockchain while users retain custody of their assets. Its sub-millisecond trading system produces more than 150 million event rows per hour, creating a need for historical indexing, real-time analytics, and operational monitoring beyond the blockchain’s state-machine model. ClickHouse Cloud consumes events from Kafka, stores them in dedicated tables, and supports user features, leaderboard and reward calculations, P&L charts, audit logs, and Grafana-backed alerts. Replacing Databricks reduced query latency from 10–15 seconds to milliseconds, improved data freshness from 1–2 hours to under 5 seconds, and removed a separate DynamoDB serving layer at comparable cost while handling 1,000 times more data across staging, testnet, and mainnet.


### [Shipyard: How We Built Slack’s Next-Generation EC2 Platform](https://yomu.fyi/post/shipyard-how-we-built-slack-s-next-generation-ec2-platform.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Archie Gunasekara
- Published: Jul 14, 2026
- Type: Problem & solution

Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows.


### [How Insider One uses ClickHouse Cloud to power real-time customer engagement at scale](https://yomu.fyi/post/how-insider-one-uses-clickhouse-cloud-to-power-real-time-customer-enga.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jul 13, 2026
- Type: Problem & solution

Insider One’s Unified Customer Database (UCD) unifies event data, user attributes, identity resolution, and profiles across channels to support real-time segmentation, personalization, and analytics. The platform initially ran ClickHouse in a self-managed AWS deployment, but growth brought larger datasets, more concurrent queries, and increasing work tuning clusters and balancing performance against cost. In a proof of concept, the team migrated an anonymized dataset from one of its largest customers to ClickHouse Cloud and replayed UI segmentation, audience-count, and scheduled-campaign workloads. Average query latency fell 50–60% without significant changes, reaching 70–75% for the largest customers, while latency percentiles became more predictable; the managed service also reduced operational burden and gave the team more capacity for product development and its AI-oriented roadmap.


### [Introducing Atlas Gen2 on AWS and GCP for M30+ Dedicated Clusters](https://yomu.fyi/post/introducing-atlas-gen2-on-aws-and-gcp-for-m30-dedicated-clusters.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Published: Jul 7, 2026
- Type: Announcement

MongoDB Atlas introduced Gen2 for M30+ Dedicated Clusters on AWS and GCP, addressing the common inefficiency of overprovisioning storage to meet IOPS requirements. Gen2 runs on newer ARM-based processors, delivering a statistically significant 15-20% reduction in CPU utilization and approximately 45% lower read and write latency in internal pilots. It introduces extended standard IOPS that can be provisioned independently from storage capacity, allowing teams to scale performance without adding unused storage and gain more precise control over infrastructure costs. No application changes or migration are required; teams retain the same Atlas API, connection strings, and operational model while benefiting from newer infrastructure underneath. Gen2 is available in supported AWS and GCP regions across APAC and Europe, with limited U.S. availability on GCP, and existing customers can evaluate upgrade opportunities for eligible M30+ clusters.


### [How Artemis Security runs 69x faster detection queries with ClickHouse Cloud](https://yomu.fyi/post/how-artemis-security-runs-69x-faster-detection-queries-with-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jul 1, 2026
- Type: Benchmark

Artemis Security uses ClickHouse Cloud on AWS for real-time threat detection across many terabytes of compressed logs, tens of billions of daily rows, and queries over trillions of rows. To reduce repeated scans, its engineers coalesce hundreds of detection-rule predicates into one query, tagging matching rows by rule ID; batches of more than 100 rules now run in under a second, with a measured 69x speedup. A Claude Code skill called /poke connects ClickHouse system tables, CloudWatch, Datadog, application data, and source code to investigate infrastructure issues through natural-language prompts. For investigative queries over semi-structured JSON, Artemis extracts frequently queried fields into a parallel materialized table and uses a hybrid two-step lookup, producing 30–60x faster queries and up to 400x lower CPU consumption.


### [Clever ingests 200x more logs at the same cost with ClickHouse Cloud](https://yomu.fyi/post/clever-ingests-200x-more-logs-at-the-same-cost-with-clickhouse-cloud.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 30, 2026
- Type: Problem & solution

Clever, an identity platform for K-12 education, faced a fragmented logging setup as its 400 services and 200 AWS Lambdas generated about 150 TB of uncompressed logs monthly. Datadog indexed only about 10% and retained it for three days, while the remainder was in Athena and required SQL. Clever migrated to ClickHouse Cloud on AWS, indexing all logs with 60-day retention and reporting 200× more searchable log volume at no additional cost, with 10× compression. Its design extracts frequently queried JSON fields into columns, stores remaining attributes in a map alongside raw JSON, uses skipping indexes, date partitions, a 60-day TTL, and reverse Unix timestamp ordering; this ordering produced an 8–10× query speedup. A custom query layer translates LogQL to SQL, while separate development and production services and multi-region deployment address isolation and availability.


[Newer posts](https://yomu.fyi/topic/aws/page/3.md) · [Older posts](https://yomu.fyi/topic/aws/page/5.md)