Loading…
From ranking to recommended: get your site ready to thrive in the age of AI agents
CloudflareMatthew Conroy
Summary
Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers.
Context
Fewer than half of HTML page requests come from humans, and customer discovery increasingly occurs through AI assistant responses rather than standard search engine results pages. Site owners previously lacked visibility into whether AI bots could navigate their sites or if models were recommending their products to users.
Approach / What changed
Cloudflare added Agent Readiness diagnostics to check technical machine-readability factors and an Answer Engine Optimization tool to query AI assistants via Cloudflare AI Gateway and evaluate citations with Workers AI. The system pre-computes industry benchmarks and surfaces traffic metrics, citation rates, and competitor share of voice in the Cloudflare dashboard.
Takeaways
- Diagnostics assesses sites across categories including basic crawler rules, API catalogs, authentication standards like MCP and Web Bot Auth, and emerging payment protocols such as x402 and ACP.
- The Answer Engine Optimization tool pre-computes category-level assistant responses to provide instant benchmarking metrics like Citation Rate, Mention Rate, Prominence, and Share of Voice.
- AI Operator Activity tracks crawl and referral volume per operator, allowing site owners to detect crawler traffic that consumes content without directing referrals back to the site.
Related reading
Introducing Radar Researcher: An AI tool for exploring Internet data in plain language
Cloudflare launched the beta of Radar Researcher, an AI tool designed to let users explore global Internet traffic data using plain-language queries. Built entirely on Cloudflare's developer platform, the system uses the Cloudflare Agents SDK within a Cloudflare Worker, maintaining conversation state and SQLite databases inside Durable Objects. The inference layer relies on Workers AI running models like Kimi K2.7 through an ordered fallback chain across three model families to prevent outages, routed via AI Gateway for logging, caching, and guardrails. Radar's frontend renders retrieved data using its existing native interactive charts, while smaller dedicated models generate conversation titles and follow-up suggestions asynchronously. Additionally, Radar introduced support for the WebMCP standard, exposing imperative and declarative tools to allow external browser agents to navigate and interact with the platform directly.
Lai Yi OhlsenBuilding an open Agentic Internet: readable, discoverable, callable, and payable
The modern web produces billions of bot requests that repeatedly fetch unchanged pages, highlighting an architecture fundamentally misaligned with automated visitors acting on behalf of paying users. Cloudflare outlines an open Agentic Internet structured around four foundational capabilities: readability, discoverability, callability, and payability. Identity and access rely on open specifications such as Web Bot Auth and Private Access Control Tokens, while Markdown for Agents and the Kitesurf browser reduce token overhead and context window bloat. For execution and monetization, WebMCP exposes structured tools directly through the browser, and protocols like x402 paired with Monetization Gateway enable per-fetch transactions. These open standards allow domain owners to define tool contracts, verify bot identities, and monetize agent traffic without relying on legacy advertising models.
Jack GalileeSay it once: introducing Bot Preference Sync
Cloudflare announced Bot Preference Sync, a feature across all customer tiers that automatically reflects dashboard AI bot configurations directly in a site's robots.txt file. Website operators previously faced challenges synchronizing static robots.txt directives with edge enforcement rules, which led some crawlers to ignore stated preferences or attempt bypasses. Under this system, Cloudflare generates and prepends directives for Search, Agent, and Training categories to existing robots.txt files using crawler lists tracked in BotBase. Mixed-use crawlers performing both search indexing and training must meet strict transparency criteria, including honoring opt-outs and providing page-level visibility, to avoid edge blocks when training is disallowed. Additionally, onboarding workflows allow ad-supported publishers to automatically set AI model training to Disallow by default while retaining search visibility.
Jin-Hee LeeUnveiling good and bad behaviors on the Agentic Internet
Web traffic increasingly shifts between human interaction and automated agents, rendering point-in-time security checks ineffective against sophisticated automation. Cloudflare addresses this shift through continuous behavioral analysis designed to evaluate trust across entire user sessions rather than relying on static point-in-time hurdles. Its CDN-injected client-side system, Precursor, tracks continuous behavioral indicators like cursor movements to catch mid-session automated takeovers. To counter rapid bot adaptation, Cloudflare is introducing self-adjusting machine learning models under Adaptive Intelligence and non-deterministic defenses such as the AI Labyrinth. These tools allow site owners to poison scraper training data, trap crawlers in synthetic webs, or queue legitimate agents while dynamically penalizing abusive traffic.
Jin-Hee Lee