Loading…
Topic
205 posts about AWS. Every summary links to the original.
See the idea
AWS Glue runs serverless Spark jobs that transform data in batches; engine-native pipelines transform during ingestion. This comparison shows how to pick between them by freshness, cost, and data fidelity.
Amazon Bedrock is the managed reasoning tier in an agent architecture, not the whole system. This layered view shows the pipelines, governed tables, query layer, and guardrails around it.
CloudFormation is a declarative reconciler: you describe the target state in a template, and it provisions, updates, and reports on the stack that results. Here is the loop.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Dynata, a large first-party data company, sought to accelerate delivery across distributed teams while improving cross-functional alignment. They adopted a staged AI approach: first AI-assisted coding via Amazon Q Developer, then connecting AI to project context from Jira, Confluence, and Lucid, and finally implementing the AI-Driven Development Lifecycle (AI-DLC) methodology using Kiro IDE. AI-DLC, powered by Amazon Bedrock, structures delivery into Inception, Design, and Construction phases with Specs, steering files, and hooks. This process forced all teams to agree on requirements and design during a Day 1 workshop, surfacing gaps before coding began. Consequently, features that once took 3–4 sprints were completed in about one sprint, cutting development phase time by roughly 70% and reducing post-launch defects.
CostBench streamed 113.2 billion stock-market quotes into both systems while four aggregate queries ran every ten minutes and two drill-down queries ran every hour. ClickHouse asynchronous inserts order raw data and build pre-aggregations during ingestion, while Redshift relies on separate background sorting and materialized view refreshes. ClickHouse Cloud led in preparation cost ($28.69 versus 522.03), normalized query cost ( 0.05 versus $24.97), and accumulated query runtime (58.12 seconds versus 1.69 hours). The combined score shows ClickHouse Cloud delivered 1,996× better end-to-end performance per dollar in this continuous-ingestion comparison. The difference originates in the preparation path: ClickHouse makes data query-ready during inserts, whereas Redshift’s streaming, sorting, and refresh steps advance independently.
Databricks announced Private Network Gateway (PNG) in public preview on Azure and private preview on AWS, addressing the networking friction that prevented teams from moving workloads to serverless compute. PNG provides a single managed gateway from serverless compute into a customer’s virtual network, eliminating the need for per-resource private endpoints, load balancers, and proxy VMs. Customers configure the gateway via a Network Connectivity Configuration (NCC) and delegate one subnet, with Databricks managing the gateway infrastructure while customers retain control over routing and security. PNG supports use cases such as ingesting from private databases via Lakeflow Connect, connecting AI agents to private MCP servers, using private package repositories, and routing egress through customer firewalls. The gateway offers two traffic modes for precise control or broad egress inspection and works alongside existing private connectivity without disruption.
Business email compromise (BEC) uses social engineering and impersonation rather than malicious links, bypassing traditional secure email gateways that are blind to internal traffic and lateral movement. VIPRE developed Integrated Email Security (IES), a cloud-native solution on AWS that integrates directly with Microsoft 365 via APIs, eliminating the need for MX record changes. The solution uses multiple machine learning layers, including semantic similarity models, contextual language models, and dynamic behavioral analysis of links and attachments, running on AWS services such as Amazon ECS with NVIDIA GPUs. In production, VIPRE IES processes over half a million messages daily with P95 latency between 1.4 and 78 seconds depending on sandbox analysis, detecting approximately 1% of messages as BEC attempts. Deployment can be completed in minutes rather than days, providing immediate visibility into internal risks that traditional gateways miss.
The Transformer architecture’s memory cost scales with token count, making long-context AI agents expensive. In controlled comparisons of three state space models - Gated KalmaNet, Gated DeltaNet, and Mamba-2 - Gated KalmaNet performed best, offering variable-length inference without retraining. Priming identifies which attention layers can be replaced by state space models using a sliding-window attention proxy, then reuses pre-trained weights to convert them into hybrid architectures without retraining, reducing pretraining cost by over 99%. A 32-billion-parameter hybrid model matches the original Transformer’s reasoning quality while decoding up to twice as fast or reaching correct answers in two-thirds of wall-clock time. The team extended hybrid context to 256K tokens without additional training and released primed models at 8B and 32B scales along with training code, kernels, and a vLLM plugin under Apache 2.0.
Antimicrobial resistance, estimated to cause 1.27 million direct deaths in 2019, requires surveillance that normally demands bioinformatics skills to turn raw reads into answers. It ingests public SRA data via a Step Functions workflow and Fargate task, then runs a HealthOmics Nextflow pipeline that trims reads, assembles them, and detects resistance genes with AMRFinderPlus using a baked-in database. Results are normalized into the hAMRonization schema and merged idempotently into Iceberg tables on S3 Tables, so reprocessing an isolate updates rows rather than duplicating them. A Strands agent on Amazon Bedrock AgentCore answers plain-language questions by generating Athena queries, with Guardrails keeping responses grounded. The solution is defined entirely in AWS CDK, deploys with a single command, and avoids manual bioinformatics steps, remaining inexpensive when idle.
iSAM Funds, a UK-based systematic asset manager, built an options research platform on ClickHouse Cloud to handle the 10,000x data fan-out from underlying instruments to related options. Migrating from Postgres, the team achieved 35-50x compression and increased single-core ingestion from 8,000 to 800,000 rows per second, while ClickHouse’s native Apache Arrow support eliminated encoding reshuffling. ClickHouse’s columnar storage and vectorized execution made queries fast enough to run on demand, allowing the team to drop pre-computed rollups and materialized views and remove a layer of pipeline complexity. The managed service integrates with iSAM’s AWS footprint and enterprise authentication, pushes backups to iSAM’s own buckets for developer use, and lets them scale instances up for large backfills then down to control costs. The platform gives researchers fast, interactive access to full options history, shortening the time from hypothesis to tested strategy.
The new AWS Digital Sovereignty Well-Architected Lens offers extended guidance for designing, building, and operating sovereign workloads, addressing challenges not fully covered by existing guidance. It includes procedures to deploy Oracle Database on Amazon EVS with FSx for ONTAP, covering provisioning, NFS mounting, and SnapMirror replication. It details multi-day AZ evacuation drills with ARC Zonal Shift across ECS, EKS, RDS for PostgreSQL, and Aurora PostgreSQL, providing CLI commands and observability metrics. MHK built the SmartProminence AI Orchestrator on Amazon Bedrock, a HIPAA-eligible framework that reduced manual medical review by 90% and cut deployment from months to weeks. CSIRO developed Serverless Beacon for genomic variant querying on AWS, implementing the GA4GH standard with S3, Lambda, DynamoDB, and Athena.
Enterprises with existing VMware Cloud Foundation investments can migrate Oracle databases to AWS using Amazon Elastic VMware Service (EVS) and Amazon FSx for NetApp ONTAP without rearchitecting applications or retraining operations teams. The architecture splits storage between vSAN for low-latency transient workloads and FSx for ONTAP NFS datastores for Oracle data files and redo logs, delivering sub-millisecond latency and up to 80,000 IOPS. Cross-region disaster recovery uses SnapMirror to asynchronously replicate FSx for ONTAP volumes, with pre-provisioned standby Oracle VMs that mount replicas only after failover to avoid Oracle double licensing. Recommended EC2 bare metal instance types include i7i.metal-24xl for per-core compute speed and i7i.metal-48xl for vertical scaling, both supporting VCF 9.1. Migration options include VMware HCX for live migration, SnapMirror for ONTAP-to-ONTAP transfer, and RMAN for universal database movement.
Teams building multi-Region architectures on AWS need to know which services are available per Region, but want that data deployed inside their VPC under their governance. This post introduces Capability Insights for AWS, a self-hosted dashboard that auto-refreshes Regional availability data every 24 hours, and Workload Analysis, which scans CloudTrail and CloudFormation to filter services to only those your account uses. Capability Insights deploys a static website backed by API Gateway, Lambda, and EventBridge within your VPC, pulling data from the centralized AWS Capabilities by Region S3 bucket. Workload Analysis reduces the scope from over 200 services to typically 20-30, attributing each resource type to the specific CloudFormation stacks that deploy it and showing property configurations in use. The dashboard provides a targeted view of service availability, API operations, and CloudFormation resource types per Region, enabling focused gap analysis for Regional expansion planning and helping identify service-availability gaps.
Enterprises centralize authentication into a single Auth0 environment but must comply with frameworks like PCI-DSS, HIPAA, and SOC 2 that restrict data to private channels. Auth0 Outbound PrivateLink provides a secure unidirectional connection between Auth0 and AWS infrastructure, avoiding the public internet. However, a single PrivateLink connection per environment cannot directly reach multiple isolated VPCs for staging and production. The Intelligent Gateway Pattern solves this by deploying an API gateway or reverse proxy at the PrivateLink termination point to fan out traffic. Auth0 Actions append context such as custom headers like X-Target-Env or specific paths, and the gateway inspects these to route requests to the appropriate isolated VPC, decoupling secure transport from internal routing. This pattern unblocks identity consolidation by allowing a single Auth0 tenant to serve both staging and production workloads while maintaining strict VPC isolation.
Datalex, a leader in airline ecommerce, needed to modernize its mature EJB2/Java 8 system for Modern Airline Retailing while maintaining continuous airline operations. Over three days, 16 Datalex engineers and 6 AWS specialists proved migration feasibility by extracting a Reservation service into a Spring Boot microservice on Java 21. The team used agentic AI tools—Kiro and AWS Transform Custom—to automate code changes, and established repeatable patterns for DevSecOps, observability, and a multi-agent AI prototype via Amazon Bedrock AgentCore. A business service proxy using the Strangler Fig pattern enabled incremental migration with backward compatibility. The board approved significant investment, the prototype team expanded into a full Agile team, and the EBA shortened discovery from an estimated 8–12 weeks to three days.
Static interfaces cannot adapt when generative AI produces variable outputs, such as a medical scan showing one fracture versus twenty ambiguous regions requiring expert review. The solution combines three AWS technologies: the AG-UI protocol for standardized agent-to-UI streaming via Server-Sent Events, the Strands Agents SDK Swarm pattern for peer-to-peer agent collaboration, and Amazon Nova Act for browser-based legacy system integration. AG-UI removes custom WebSocket formats by defining typed events that both agent frameworks and frontends implement as a universal contract. Strands swarm agents debate hypotheses and iteratively refine findings, producing explainable results visible in real time, while Nova Act navigates login screens and fills forms on systems without APIs. A complete radiology assistant walkthrough demonstrates reduced interface development time and HIPAA-compliant data handling for medical imaging.
Reactor and the AWS Neuron Science team collaborated to optimize real-time video generation on AWS Trainium. They focused on autoregressive diffusion models, which combine sequential next-token prediction with iterative diffusion to generate video frames that respond to user input. The teams used the Rolling Forcing model as a proxy, addressing challenges such as dynamic shapes, unusual memory access patterns, and heavy cache management that generic compilers struggle with. Optimizations included splitting SDPA heads across four cores and sequences across two, and batching common components of the diffusion and cache update phases to improve hardware utilization. The collaboration achieved real-time video generation above 16 frames per second on a single Trainium2 chip, a result that generalizes across autoregressive diffusion models.
ReadyOn describes a four-layer tenant-isolation architecture for Harmony, its multi-tenant workforce-intelligence platform on Amazon EKS, which handles payroll data, organizational hierarchies, and operational analytics. The design combines dedicated Kubernetes namespaces, Karpenter-managed tenant node pools, per-tenant Amazon VPC security groups, and dedicated Amazon Aurora databases, with RBAC, admission control, default-deny network policies, IAM Roles for Service Accounts, and tenant-scoped secrets. GitOps through Argo CD ApplicationSets provisions identical controls and continuously reverts drift, while dual taints and admission validation prevent cross-tenant scheduling. ReadyOn says adversarial exercises have blocked all six tested cross-tenant paths to date, and reports defenses across 10 threat-model stages, though shared control-plane dependencies make the layers defense in depth rather than fully independent probabilities.
CSIRO built Serverless Beacon (sBeacon), a production-ready implementation of the Global Alliance for Genomics and Health (GA4GH) Beacon protocol for secure genomic variant querying. Its serverless architecture combines Amazon S3, AWS Lambda, Amazon DynamoDB, and Amazon Athena: genomic data remains in its original S3 location, while metadata is indexed in ORC tables and ontology indexes for query processing. Standard VCF files can be queried through API Gateway, with modular Lambda functions resolving ontology terms, querying Athena, and, when needed, issuing HTTP byte-range requests to source buckets so raw VCF bytes do not leave their owning organization. The post reports support for hundreds of millions of individuals and billions of genomic locations, approximately USD 0.40 monthly operation for a 1000 Genomes-scale dataset, roughly five-second real-world queries, and about 18-second generation-to-use onboarding, while noting that whole-genome range queries can time out.
Addepar, a global technology and data platform, manages nearly $9 trillion in assets and moved from older systems to a unified data intelligence platform on Databricks running on AWS. It ingests hundreds of disparate data feeds, standardizes and enriches them, and delivers results through products, APIs and data sharing. Since adoption, pipeline costs fell 60% versus legacy infrastructure, saving more than $2 million, while new pipelines and integrations became five times faster. Addison, a native AI experience embedded in the platform, combines governed portfolio Q&A, proactive insights and action-oriented workflows, using live data and Addepar’s core calculations engine. Addepar is adopting Databricks Agent Bricks, including Supervisor Agent flows with Genie-powered analytics, while Foundation Model Serving and managed MLflow support governed model serving, traceability, evaluation and iteration for human-supervised agentic workflows.
Multi-hospital networks face rising imaging volumes, siloed on-premises PACS, radiologist routing bottlenecks, and storage that must expand for 7 to 10 years of retention. The proposed pattern uses a hub-and-spoke hybrid architecture: local PACS instances preserve LAN-speed reading, while a centralized AWS archive provides cross-facility discovery, disaster recovery, and lifecycle-managed storage. Modalities send DICOM through C-STORE to local VNA servers, which normalize and compress objects; asynchronous replication sends images to Amazon S3, while Amazon Aurora PostgreSQL stores metadata and routing information. The AWS deployment spans two Availability Zones, places PACS services behind Network Load Balancers, and includes CloudFront and AWS WAF. Storage guidance distinguishes S3 Intelligent-Tiering, Glacier Instant Retrieval, and Glacier Deep Archive according to access frequency and retention needs.
DHI Group needed to move generative AI efforts beyond proofs of concept because its traditional software development lifecycle took months, while it sought faster delivery, technical validation, and broader AI literacy. AWS and DHI used a four-phase Hackathon Acceleration Package—preparation, enablement, a three-day build event, and a path to production—with workshops on Amazon Bedrock AgentCore, AI-DLC, and Kiro. Twenty participants formed three teams and produced prototypes for employer analytics, intelligent candidate matching, and a unified ClearanceJobs–AgileATS marketplace; DHI leadership committed to advancing all three toward production. The winning marketplace uses an MCP server, Amazon Bedrock AgentCore, Lambda functions, session memory, and Claude 3.5 Haiku to let recruiters search candidates and enrich profiles through natural-language requests. The post concludes that time-boxed, cross-functional hackathons can compress ideation, architecture, prototyping, executive alignment, and production planning when production criteria and preparation are established upfront.
This post presents an inventory automation architecture that moves from zero-shot demand forecasts to validated purchase orders while applying safety stock, minimum order quantities, budgets, and promotional adjustments. It combines Amazon Chronos2, hosted on SageMaker Serverless Inference, with four Claude-powered agents built using the Strands Agents SDK and deployed on Amazon Bedrock AgentCore; deterministic Python tools perform computation and agents handle judgment. Product data and future covariates arrive through CSV files in S3, while business rules remain in JSON, enabling what-if forecasts and onboarding without per-SKU training. In internal testing across 50 SKUs over a four-week horizon, the system reached 12.3% median WAPE, reduced onboarding from 2–3 weeks to under five minutes, and lowered estimated monthly inference cost from about $1,091 to $15. A conditional retry loop surfaces budget violations for user decisions instead of silently truncating orders, with traces and evaluator scores supporting auditability.
The best engineering reads, once a week.
