API Versioning as a Translation Layer
API versioning as an architecture: requests enter through a version header, pass through a chain of reversible version changes, and reach controllers that only ever see the latest model.
Loading…
Topic
Architecture
830 posts about Architecture. Every summary links to the original.
- 830 posts
- 50 companies
- 31 explainers
See the idea
Architecture explainers
BIAN as a Contract Layer: Modernizing Core Banking One Domain at a Time
BIAN gives banks a shared vocabulary and service-domain contracts that sit between business concepts and the systems implementing them. See how domains, contracts, and governed access stack into one architecture.
Argo CD: The Reconciliation Loop Between Git and Your Cluster
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Sciene AI Companion: building an autonomous Customer Success platform on Databricks
Sciene built AI Companion for Quartile’s Customer Success organization, where CSMs support more than 1,000 brands and previously spent substantial time preparing decks, reconstructing context, and investigating account changes. The platform addresses personalization at scale, high-volume content generation, and root-cause diagnosis by combining account data, CSM communication styles, company principles, and cross-domain business data. Its Email Hub cuts reply time from 15–30 minutes to about three minutes, Meeting Hub reduces preparation for 80+ slide decks from over two hours to around 10 minutes, and Account Flagging reduces diagnosis of flagged accounts from 30+ minutes to about five. Databricks provides the shared governed foundation: Delta Sharing supplies data without copies, Lakebase stores operational state, and SQL Warehouses serve analytical, AI, and operational workloads from the same tables. The design keeps CSMs responsible for judgment while giving them current context for customer interactions.
Announcing Lakebase Search: agent-native retrieval built into Lakebase Postgres
Lakebase Search is a beta offering on AWS and Azure that adds hybrid vector and full-text retrieval to Lakebase Postgres. It uses the lakebase_vector and lakebase_text extensions to keep retrieval, memory, operational data, and hybrid search in one backend. lakebase_vector retains pgvector types and operators, applies RaBitQ clustering and compression for 32x smaller indexes, and targets more than 1B vectors, while lakebase_text replaces GIN with object-storage-optimized BM25 ranking. A tiered cache keeps hot data on NVMe and places colder data in object storage; the source reports lower memory needs, faster index builds, and cold-cache startup than standard pgvector HNSW in its LAION-100M benchmark. The extensions also combine vector similarity and keyword relevance with reciprocal rank fusion in a single SQL query, enabling joins and tenant filtering alongside transactional workflows.
From hobby to real world: integrating the hard stuff with agents
The post documents building a production-oriented leaderboard for open-source Stripe Projects applications, where users submit projects, share links, and vote. Unlike an earlier transcription app, this project required authentication, multiple database tables, hosting, and controls limiting unauthenticated users to one vote per project. The agent initially selected Supabase for database and auth and Vercel for hosting, then added Upstash Redis after an in-memory rate limiter proved unsuitable for serverless enforcement. Supabase's two-emails-per-hour authentication limit prompted a human-directed migration to Clerk, after the agent proposed an unsuitable Twilio solution. The resulting MVP used five resources across four providers, took more than a day, and received separate security and performance audits before review.
Cisco Talos empowers threat researchers while reducing TCO by 75% with ClickHouse Cloud
Cisco Talos operates a threat-intelligence reputation service that classifies file hashes from Cisco product telemetry, enabling researchers to make security decisions about benign and potentially malicious artifacts. Previously, a custom pipeline moved data from Databricks Delta and S3 through EKS, where staging and aggregated tables supported an API; the self-managed design retained about a petabyte of compressed, replicated data on 32 high-compute nodes. The team rebuilt the system in four phases, using Terraform, SSO/SAML, sharded ClickPipes for a 12-month backfill, shadow traffic, and a validated cutover to ClickHouse Cloud on AWS. The resulting architecture uses EventBridge, SQS, ClickPipes, SharedMergeTree, object storage, and AWS PrivateLink, delivering less than a minute of queue lag, zero data loss, a 100% response match across more than 1,000 hashes, zero downtime, roughly 90% lower storage costs, and about 75% lower TCO while reclaiming over 300 engineer hours annually.
How API changes flow into Stripe's developer products
Stripe describes how API changes are propagated across its developer-product suite, including SDKs in seven languages and three release channels, the CLI, Shell, API Explorer, API Reference, changelog, and internal tools. Historically, v1 workflows used OpenAPI or a Ruby-based DSL while v2 APIs used protobuf-based DSLs and an internal API specification, forcing teams to maintain parallel generators and leaving some products without v2 support. Stripe chose OpenAPI as a shared representation and built a pipeline whose generated artifacts are tailored by variant, release phase, and API version, with vendor extensions carrying Stripe-specific metadata. Automated validation, OpenAPI diffs, pre-release comparisons, versioned snapshots, and downstream publishing now connect API definition changes to documentation, code samples, SDK and CLI releases, while older v1 versions remain difficult to regenerate with newly added metadata.
Introducing Omnigent: A Meta-Harness to Combine, Control and Share Your Agents
Omnigent is an open-source meta-harness designed to combine, control, and share agents across different harnesses, models, and interfaces. Databricks says users currently juggle multiple agents and copy context between them, while builders struggle to combine or replace harnesses with incompatible interfaces. Its runner wraps terminal-based agents and SDKs in sandboxed sessions with a uniform API for messages, files, streamed text, and tool calls; a server adds policies, sharing, and access through terminal, web, mobile, native Mac OS, and APIs. Features include live collaboration, hosted sandbox execution, contextual security and cost policies, OS isolation, and multi-harness authoring. Released in alpha under Apache 2.0, Omnigent is intended to provide a durable layer above changing agents and harnesses.
Enabling Evolutionary Database Development: Database branching with Lakebase, the conclusion
The post concludes a series on how copy-on-write database branching in Databricks Lakebase changes team-scale evolutionary database development without changing its underlying methodology. For a team of fifty developers, long-running tier branches and ephemeral feature branches form a parent-linked promotion hierarchy, replacing separately provisioned environment instances and enabling promotion by merge, rollback by repoint, and computable schema divergence. Governance is declared once and inherited per branch, with policies intended to prevent transitions that contradict the parent chain; Unity Catalog captures metadata for attribution and audit. The DBA's role becomes platform engineering, while agents operate inside an executable SCM state machine with documented inputs, outputs, schema validation, and enforced gates. An optional TDD layer adds dedicated roles, acceptance-criterion scenarios, RED-GREEN-REFACTOR cycles, and artifact contracts, and the conclusion presents the resulting workflow as operational for human and agent practitioners.
Unlocking semantics for AI: How Mercedes-Benz Korea built trusted “Talk to Data” at scale
Mercedes-Benz Korea piloted a “Talk to Data” architecture that extends its Databricks analytics foundation with a governed semantic layer for enterprise AI, rather than treating the effort as a chatbot project. The design moves Power BI DAX KPI logic into Unity Catalog Business Semantics and Metric Views, keeping sources, joins, measures, dimensions, comments, and synonyms alongside governed Lakehouse data. Genie spaces use curated metric views for domain questions, while Agent Bricks composes persona-based agents, with Unity Catalog enforcing row- and column-level access. An automated DAX-to-Metric-View transpiler parses semantic models, maps tables, generates draft definitions, flags non-automatable measures, and reports conversion gaps. The documented playbook combines gold-layer curation, KPI validation, regression testing, Genie optimization, persona agents, and Databricks Apps; the pilot reports AI answers aligned with established KPI definitions and BI reporting logic.
Production-Ready Agents Need A Production-Ready Data Platform
AI development teams face constant shifts in model providers and agent frameworks, demanding data platforms that provide scalable, real-time context management. Agentic workloads require blending unstructured enterprise data, short-term session state, and persistent long-term memory. MongoDB addresses these requirements through its native JSON document model and integrated retrieval capabilities, combining full-text search, vector search, and hybrid search directly over operational data. Customer implementations such as DevRev, ElevenLabs, and Adobe rely on Atlas to achieve sub-100 millisecond hybrid retrieval and handle billions of requests. Additionally, MongoDB is collaborating with LangChain and ecosystem partners to establish open reference architectures and shared interfaces for portable agent memory across frameworks.
Simplifying Data and Product Integrations with a Data Abstraction Layer
Uber built a domain-agnostic Data Abstraction Layer (DAL) to decouple data consumers from changing physical datasets and simplify advertiser reporting. The DAL is an RPC service that accepts a logical table, schema, time range, and filters; logical tables define the interface, while configuration describes locations, relationships, availability windows, and rollup transforms. It generates database-specific queries, runs them in parallel across systems such as an internal OLAP database, Docstore, and Apache Hive, and assembles responses with aggregation and rollup behavior determined by the request. Removing a dimension can produce an account-level response with metrics automatically rolled up, while mixed realtime and daily tables can cover different portions of a requested time range. For advertiser reporting, the approach reduced new report delivery from several weeks or months to under two days and enabled dynamic filters, report builders, and third-party API integrations.
Stop building data products. Start building data services.
Howden’s rapid acquisition pace exposed limits in an enterprise data model built around one product per use case, downstream quality checks, and dashboard-driven consumption. Group Chief Data Officer Barry Panayi describes shifting to open, governed data services, moving mastering and quality checks closer to ingestion, and codifying reconciliation in the Accord data model. On Databricks, the company consolidated more than 100 sources of record, standardized pipelines and shared code, and built reusable assets for cross-domain analytics, while continuing to productionize models as consistent services. The account argues that AI agents require a composable services layer, and that insight lag—the time between data existing and being usable—matters more than freshness; conversational analytics through Genie also reduced dashboard-building work.
Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions
As Lyft scaled, different teams developed conflicting definitions for key business metrics due to the lack of centralized version control and shared standards. To resolve this, Lyft implemented an internal Metric Semantic Layer as a Python package that stores authoritative metric definitions in YAML files with Jinja SQL templates. The system restricts onboarding to Golden Metrics used across multiple applications, requiring team-based approval from both Business and Operational Owners for any definition changes. Standardized definitions are exposed through Python APIs, integrated into the Amundsen data catalog and self-service user interfaces, and surfaced to AI tools via a Model Context Protocol.
amazon.science
EC2’s formally verified “isolation engine” provides mathematical assurance of virtual-machine isolation
AWS has introduced M9g and M9gd EC2 instances powered by Graviton5 CPUs, featuring the formally verified Nitro Isolation Engine to enforce virtual machine isolation. To prove the engine's correctness, engineers developed 330,000 lines of machine-checked mathematics using the Isabelle/HOL proof assistant. The system architecture separates hypervisor policy from isolation mechanisms, requiring the deprivileged Nitro Hypervisor to route all guest state modifications through the minimal Rust-based engine. Functional verification uses a restricted language subset called micro Rust, Separation Logic contracts specifying total correctness, and weakest-precondition calculus from the open-source AutoCorrode library. Finally, high-level transition relations and noninterference proofs ensure that memory integrity and guest confidentiality are preserved across every execution step.
amazon.science
Graviton5’s improved design increases speed and energy efficiency — beyond Moore’s law
Amazon announced the general availability of M9g and M9gd EC2 instances powered by the new Graviton5 processor. Built on a three-nanometer process, Graviton5 features 192 Neoverse V3 cores, DDR5-8800 memory support, PCIe gen6 interconnects, and 192 megabytes of level-three cache. The chip transitions from seven discrete dies to four unified chiplets connected at 420 gigabytes per second, eliminating dedicated I/O and memory controller dies. These architecture updates deliver up to 25% higher computational performance over Graviton4, with up to 35% faster performance for web applications and machine learning inference. The new instances also introduce the Nitro Isolation Engine, which uses formal verification to enforce hardware and software isolation between virtual machines.
Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant
Spotify developed an AI data assistant, Vedder, to scale access to over 70,000 internal datasets without overwhelming domain experts. Because raw warehouse schemas omit critical semantic nuances and overwhelm LLM context windows, engineers implemented a domain-owned context layer organized into clusters. Each cluster contains profiled datasets, expert-curated business documentation, and vetted question-and-SQL pairs that guide a ReAct-based agent. Automated query history extraction proved insufficient on its own, as domain curators accepted only 12.5% of inferred query pairs due to noise. Deployed across Slack, IDEs, and a dedicated web interface, the system maintains reliability through continuous cluster health monitoring that flags schema drift and degraded examples.
Quick: An internal hosting platform for the AI era
Shopify built Quick to make internally created prototypes, dashboards, tools, and games easier to share than deploy, giving employees a secure URL for a folder of HTML and assets. Launched internally in July 2025, it places sites behind Identity-Aware Proxy, serves bucket content through NGINX and gcsfuse, and deploys with a wrapper around gcloud rsync. A service in front of CloudSQL provides database and realtime APIs, while server-held credentials support calls to Shopify’s AI proxy; file uploads, BigQuery access, WebSockets, and employee identity are also available. Quick now hosts more than 50,000 sites, has been used by over half of Shopify employees, and runs on one $200-per-month VM, with constraints, rate limiting, and a migration from Node to Go supporting its operation.
Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world
To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.
How QRT powers real-time research and risk management at petabyte scale
Qube Research & Technologies (QRT), a global quantitative investment manager headquartered in London, uses ClickHouse Cloud for a data platform that serves researchers and for near-real-time risk monitoring, management, and P&L. Founded in 2018 as a cloud-native firm after the spinout of Credit Suisse’s proprietary trading arm, QRT chose a cloud-based approach rather than inheriting decades of on-premise infrastructure. The platform supports two major systems, while ClickStack consolidates logs, metrics, and traces for observability; alerts, log queries, and metric aggregations run against a single ClickHouse instance using standard SQL. QRT says its previous database was constrained by writer and reader counts and total cluster load, whereas ClickHouse provides greater speed and scalability, and that a unified monitoring backend simplifies maintenance while remaining reliably fast.
How an Agent Built a 3D Paris Gallery by Chaining Two Hugging Face Spaces
A coding agent built a 3D Paris gallery by chaining two Hugging Face Spaces: an image-generation Space produced dark-background monument images, and VAST-AI/TripoSplat reconstructed each single image as a 3D Gaussian splat. It then handled integration work by correcting the splats’ Y-down orientation, auto-framing monuments, compressing .ply files to .ksplat at about one-third the size, and building and deploying a static Three.js viewer. The Spaces’ agents.md files provided schemas, call and polling templates, file-upload instructions, and authentication guidance, allowing the agent to call them without a client library. The same pipeline produced six-monument galleries for Egypt and Japan after the prompts changed. The account presents callable, documented multimedia components as a way to reduce integration work and make new multimedia applications cheaper to create.
amazon.science
Real-world grounding in agentic AI
Foundation models deployed as physical AI agents face critical risks when natural language hallucinations translate into physical law violations and operational hazards. To ensure physical consistency and reliability, researchers from the University of California, San Diego and Amazon Fulfillment Technology introduced four grounding pillars. Physics-guided deep learning embeds symmetries and differential equations into pretraining to obey conservation laws while reducing dataset requirements. The UQ4CT framework applies mixture-of-experts architectures to produce calibrated uncertainty estimates, cutting expected calibration error by over 25% and halting operations when safety thresholds are breached. Furthermore, the Adapting-While-Learning framework and formal verifiers like Hilbert integrate numerical simulators and interactive proof systems to enforce mathematical and scientific correctness.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.