Loading…
Cloudflare AI Search: give your agents a search engine for your data
CloudflareNelson Duarte
Summary
Cloudflare introduced developer experience improvements and a preview pricing model for Cloudflare AI Search. Previously, users had to manually connect separate Cloudflare primitives like Workers AI, AI Gateway, Vectorize, R2, and Browser Run to build a search solution. The platform now automates crawling, ingestion, embedding, and retrieval across structured and unstructured data sources. Developers can index sites without sitemaps using link discovery, expose public /search and /mcp endpoints across namespaces, bind search instances directly to Workers, or secure endpoints with Cloudflare Access. Cloudflare uses this system for developer documentation, its corporate blog, and the Cloudflare Dev Stack MCP server to supply coding agents with cited documentation.
Context
Building search solutions previously required developers to stitch together multiple Cloudflare primitives manually, including Workers AI, AI Gateway, Vectorize, R2, and Browser Run.
Approach / What changed
Cloudflare automated the crawling, ingestion, embedding, and retrieval pipeline within AI Search, adding sitemap-free discovery parsing, public /search and /mcp namespace endpoints, custom domain support, and direct Worker bindings.
Takeaways
- Embedding and reranking are included for free when using default or select models from the Workers AI catalog.
- Namespaces can expose public /search and Model Context Protocol (/mcp) endpoints without authentication, which can be branded with custom domains or secured via Cloudflare Access.
- The crawler operates via Browser Run under the Cloudflare-AI-Search bot identity, respecting site bot policies and robots.txt files.
Related reading
Secure all your internal vibe-coded applications — in one click
AI-assisted development enables employees to build applications rapidly, but unmanaged deployments can accidentally expose internal company data to the public Internet. Cloudflare introduced direct Cloudflare Access integration for Cloudflare Workers, allowing organizations to enforce authentication policies directly at the account or individual Worker level rather than per hostname. When enabled, incoming requests are authenticated before reaching application code across custom domains, routes, workers.dev subdomains, and preview environments. Developers can access authenticated identity data such as emails and groups directly via the Worker context object without manually validating JSON Web Tokens. This capability was implemented on Cloudflare's Rust-based FL2 proxy, which separated Worker routing from Worker execution so routing runs safely prior to Access policy enforcement.
Chythra MalapatiBuilding an open Agentic Internet: readable, discoverable, callable, and payable
The modern web produces billions of bot requests that repeatedly fetch unchanged pages, highlighting an architecture fundamentally misaligned with automated visitors acting on behalf of paying users. Cloudflare outlines an open Agentic Internet structured around four foundational capabilities: readability, discoverability, callability, and payability. Identity and access rely on open specifications such as Web Bot Auth and Private Access Control Tokens, while Markdown for Agents and the Kitesurf browser reduce token overhead and context window bloat. For execution and monetization, WebMCP exposes structured tools directly through the browser, and protocols like x402 paired with Monetization Gateway enable per-fetch transactions. These open standards allow domain owners to define tool contracts, verify bot identities, and monetize agent traffic without relying on legacy advertising models.
Jack GalileeFrom ranking to recommended: get your site ready to thrive in the age of AI agents
Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers.
Matthew ConroyUnifying Workers AI and AI Gateway into a single AI control plane
Workers AI and AI Gateway are converging into a unified AI control plane to simplify model access, billing, and observability. Developers can now access both services through a single entrypoint using unified Workers bindings and the /ai/ REST endpoint. Specifying a default gateway automatically instruments requests with payload logging, token tracking, and cost attribution without manual dashboard setup. Cloudflare has also rolled out unified prepaid billing, allowing teams to spend AI Gateway credits on Workers AI models while unlocking higher rate limits. Planned future features include model-first routing for automated provider failover and smart routing powered by task classifiers on Workers AI.
Michelle Chen