Loading…
Building an open Agentic Internet: readable, discoverable, callable, and payable
CloudflareJack Galilee
Summary
The modern web produces billions of bot requests that repeatedly fetch unchanged pages, highlighting an architecture fundamentally misaligned with automated visitors acting on behalf of paying users. Cloudflare outlines an open Agentic Internet structured around four foundational capabilities: readability, discoverability, callability, and payability. Identity and access rely on open specifications such as Web Bot Auth and Private Access Control Tokens, while Markdown for Agents and the Kitesurf browser reduce token overhead and context window bloat. For execution and monetization, WebMCP exposes structured tools directly through the browser, and protocols like x402 paired with Monetization Gateway enable per-fetch transactions. These open standards allow domain owners to define tool contracts, verify bot identities, and monetize agent traffic without relying on legacy advertising models.
Context
Traditional web architecture, analytics, and monetization models are built for humans rendering visual interfaces and viewing ads, causing automated agents to waste compute by re-fetching unchanged content, parsing raw DOM elements, and polluting context windows with unrendered HTML tags.
Approach / What changed
Cloudflare outlines an open Agentic Internet framework based on open standards (x402, MCP, Web Bot Auth, PACT). The architecture makes sites readable via Markdown for Agents and the Kitesurf browser, discoverable via AI Search and Agent Engine Optimization, callable via WebMCP tool registration, and payable via Monetization Gateway and agent wallets.
Takeaways
- WebMCP allows websites to register explicit tool contracts directly via document.modelContext.registerTool, eliminating the need for agents to parse HTML DOM structures or synthesize clicks.
- Web Bot Auth provides cryptographic bot identity verification, while Private Access Control Tokens (PACT) enable anonymous third-party vouching across participating platforms.
- The Kitesurf browser executes on Cloudflare Workers on a per-request basis and is discarded immediately after, removing human-oriented browser bloat to optimize agent performance.
Related reading
From ranking to recommended: get your site ready to thrive in the age of AI agents
Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers.
Matthew ConroySecure all your internal vibe-coded applications — in one click
AI-assisted development enables employees to build applications rapidly, but unmanaged deployments can accidentally expose internal company data to the public Internet. Cloudflare introduced direct Cloudflare Access integration for Cloudflare Workers, allowing organizations to enforce authentication policies directly at the account or individual Worker level rather than per hostname. When enabled, incoming requests are authenticated before reaching application code across custom domains, routes, workers.dev subdomains, and preview environments. Developers can access authenticated identity data such as emails and groups directly via the Worker context object without manually validating JSON Web Tokens. This capability was implemented on Cloudflare's Rust-based FL2 proxy, which separated Worker routing from Worker execution so routing runs safely prior to Access policy enforcement.
Chythra MalapatiCloudflare AI Search: give your agents a search engine for your data
Cloudflare introduced developer experience improvements and a preview pricing model for Cloudflare AI Search. Previously, users had to manually connect separate Cloudflare primitives like Workers AI, AI Gateway, Vectorize, R2, and Browser Run to build a search solution. The platform now automates crawling, ingestion, embedding, and retrieval across structured and unstructured data sources. Developers can index sites without sitemaps using link discovery, expose public /search and /mcp endpoints across namespaces, bind search instances directly to Workers, or secure endpoints with Cloudflare Access. Cloudflare uses this system for developer documentation, its corporate blog, and the Cloudflare Dev Stack MCP server to supply coding agents with cited documentation.
Nelson DuarteThe Cloudflare Blog – Brought to you by EmDash
Cloudflare migrated its official blog to EmDash, an Astro-focused content management system, acting as internal Customer Zero to evaluate scaling and usability. To prepare for traffic spikes reaching thousands of requests per second, engineers used k6 to execute ramp, breakpoint, and burst load tests. The resulting production architecture runs EmDash on Cloudflare Workers, pairing it with Workers Cache, an object cache backed by Workers KV, and Hyperdrive connected to PlanetScale. This layered caching design serves 99.5 percent of static files and 70 percent of requests from cache, flattening p95 latency under real-world loads reaching 850 requests per second. The migration also introduced a frontend redesign using the Kumo design system and added Model Context Protocol servers for automated agent workflows.
Kody Jackson