# From ranking to recommended: get your site ready to thrive in the age of AI agents

[Cloudflare](https://yomu.fyi/company/cloudflare) · Matthew Conroy · Aug 6, 2026

**Type:** Announcement

## Summary

Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers.

## Context

Fewer than half of HTML page requests come from humans, and customer discovery increasingly occurs through AI assistant responses rather than standard search engine results pages. Site owners previously lacked visibility into whether AI bots could navigate their sites or if models were recommending their products to users.

## Approach / What changed

Cloudflare added Agent Readiness diagnostics to check technical machine-readability factors and an Answer Engine Optimization tool to query AI assistants via Cloudflare AI Gateway and evaluate citations with Workers AI. The system pre-computes industry benchmarks and surfaces traffic metrics, citation rates, and competitor share of voice in the Cloudflare dashboard.

## Takeaways

- Diagnostics assesses sites across categories including basic crawler rules, API catalogs, authentication standards like MCP and Web Bot Auth, and emerging payment protocols such as x402 and ACP.
- The Answer Engine Optimization tool pre-computes category-level assistant responses to provide instant benchmarking metrics like Citation Rate, Mention Rate, Prominence, and Share of Voice.
- AI Operator Activity tracks crawl and referral volume per operator, allowing site owners to detect crawler traffic that consumes content without directing referrals back to the site.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Search](https://yomu.fyi/topic/search)

[Read original post](https://blog.cloudflare.com/aeo)