The API gateway is the single boundary where protocol translation, auth, quotas, and routing become shared policy instead of per-service code. This layered diagram shows what belongs there and why.
An API gateway sits between callers and your services, and every layer it owns is a decision about what the platform should handle once rather than each service repeating. At the edge it terminates identity, quotas, and protocol translation, so an existing REST endpoint can be exposed as an MCP tool without standing up a second server. Above it sits a governance plane that decides which agents, models, and spend policies are approved, and below it sit the services and model endpoints it routes to. Seeing that split clearly helps you decide what to centralize, what to push behind the boundary, and where overload protection actually belongs.
Cross-cutting concerns that every caller needs — identity, quotas, protocol translation, and routing — belong at the boundary, because duplicating them per service multiplies the surface you must keep consistent. Business logic specific to one service stays behind the boundary. The architectural question is always which policy is genuinely shared.
Google Cloud API Gateway in Public Preview acts as a remote MCP server: annotate your OpenAPI spec with x-google-api-management.mcp, deploy, and the gateway accepts MCP JSON-RPC on a single /mcp endpoint. It transcodes tools/call into the corresponding REST call, runs it through existing policies, and supports up to 1,000 tools per gateway without a separate server.
Salesforce's Cloud Atlas team found per-instance rate limiting insufficient under autoscaling and bursty agent-driven traffic. Their redesign watches queue time as an early pressure signal before error rates spike, and shares customer usage between servers so noisy tenants are throttled without a central coordinator. That sheds low-priority work before overload spreads to dependent services.
The posts behind it
Databricks · · AnnouncementOrganizations face a challenge as new frontier models appear roughly every five days, making it difficult to balance flexibility with governance. Databricks introduces the Unity Gateway CLI, which lets admins centrally configure approved coding agents, models, MCP servers, skills, and spending policies. Developers launch any approved agent with a single ug command, and the CLI handles authentication and applies the published settings. Features include Smart Routing that selects cost-appropriate models per task, budget-aware defaults that switch to lower-cost options as spending thresholds are reached, and tracing to capture tool calls and skill invocations for waste analysis. In internal use, the system processed over 61 billion input tokens across roughly 360,000 requests, and tracing helped identify bugs that saved an estimated 1.2 million per year.
Salesforce · · Problem & solutionSalesforce’s Cloud Atlas team operates a globally distributed identity data store that must maintain five nines of availability, roughly five minutes of downtime per year. The team redesigned service protection after per-instance rate limiting proved insufficient under autoscaling and increasingly bursty, unpredictable agent-driven traffic. The new approach combines intelligent load shedding, which monitors queue time as an early indicator of system pressure before error rates spike, with coordinator-free global quota management. Servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, preventing one customer’s surge from affecting others. This design prevents cascading failures by shedding lower-priority work before overload spreads across dependent services, ensuring graceful degradation rather than widespread outage.
Google · · AnnouncementMost enterprise REST APIs are invisible to AI agents, forcing teams to run separate MCP servers that duplicate their existing gateway logic. Google Cloud API Gateway in Public Preview solves this by acting as a remote MCP server — annotate the OpenAPI 3.0.x/3.1.x spec with x-google-api-management.mcp and deploy it with no extra infrastructure. The gateway accepts MCP JSON-RPC requests on a single /mcp endpoint, transcodes tools/call into the corresponding REST call, runs it through existing policies, and returns the response as an MCP result. Authentication via JWT or API key works unchanged for REST operations; tools/list is unauthenticated by default and requires JWT in production for security. With no separate server to operate, teams in Public Preview can expose up to 1,000 tools per gateway that support existing authentication and quota, and the MCP server is automatically discoverable via API hub and Agent Registry.
Databricks · · ExplainerConcurrence processes 100.8 billion input tokens and 11.2 million LLM calls every 30 days, at an annualized rate of 1.2 trillion tokens. To govern these workflows, the company consolidates on Databricks with Lakebase for operational state, Unity Catalog for governance, and Unity Gateway for centralized AI access. Patient data is recorded as immutable events in Delta tables, from which a current patient state—called the world model—is computed while preserving provenance. All agents are tested against simulated patients before deployment; simulation traffic is approximately seven times production traffic, using immutable event history to replay states. Concurrence routes all coding-agent traffic through Unity Gateway, attributing each request to the engineer, and enforces compliance-gated model routing where HIPAA requirements are met first.
Red Hat · · Problem & solutionDefault inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.
More like this
AI-DLC is an AI-driven development lifecycle that runs delivery through Inception, Design, and Construction, with specs, steering files, and hooks keeping agents aligned to agreed intent.
Apple Silicon's defining choice is one shared memory pool for CPU, GPU, and Neural Engine. This diagram maps that fabric, the on-package bandwidth it buys, and the scaling limits it imposes.
Antimicrobial resistance surveillance is a layered data architecture: raw reads and demand signals flow in, get normalized into governed tables, and answer plain-language queries. The diagram maps each boundary.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
