Loading…
Say it once: introducing Bot Preference Sync
CloudflareJin-Hee Lee
Summary
Cloudflare announced Bot Preference Sync, a feature across all customer tiers that automatically reflects dashboard AI bot configurations directly in a site's robots.txt file. Website operators previously faced challenges synchronizing static robots.txt directives with edge enforcement rules, which led some crawlers to ignore stated preferences or attempt bypasses. Under this system, Cloudflare generates and prepends directives for Search, Agent, and Training categories to existing robots.txt files using crawler lists tracked in BotBase. Mixed-use crawlers performing both search indexing and training must meet strict transparency criteria, including honoring opt-outs and providing page-level visibility, to avoid edge blocks when training is disallowed. Additionally, onboarding workflows allow ad-supported publishers to automatically set AI model training to Disallow by default while retaining search visibility.
Context
Maintaining multiple layers of protection on websites is cumbersome when stated robots.txt preferences and edge enforcement rules disagree, which can lead crawlers to disregard preferences or bypass rules. Furthermore, mixed-use crawlers blending search, agent, and training behind a single user agent make it difficult for site owners to separate desired search indexing from unwanted model training.
Approach / What changed
Cloudflare introduced Bot Preference Sync across all plan tiers to automatically reflect dashboard AI bot configurations (covering Search, Agent, and Training categories) directly into robots.txt. The feature prepends generated directives to existing files and uses BotBase to update crawler lists. Additionally, bot operators performing both search and training must satisfy specific transparency criteria to access content for indexing when Training is disallowed.
Takeaways
- When site owners enable Bot Preference Sync, generated AI bot directives are prepended to the existing robots.txt file so preexisting Disallow rules remain intact.
- Mixed-use bots performing search and training must satisfy transparency criteria, such as URL-level visibility and AI summary opt-outs, to access content when training is disallowed.
- Ad-monetized publishers can select a dedicated onboarding option that defaults AI training to Disallow while maintaining search indexing.
Related reading
From ranking to recommended: get your site ready to thrive in the age of AI agents
Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers.
Matthew ConroyUnveiling good and bad behaviors on the Agentic Internet
Web traffic increasingly shifts between human interaction and automated agents, rendering point-in-time security checks ineffective against sophisticated automation. Cloudflare addresses this shift through continuous behavioral analysis designed to evaluate trust across entire user sessions rather than relying on static point-in-time hurdles. Its CDN-injected client-side system, Precursor, tracks continuous behavioral indicators like cursor movements to catch mid-session automated takeovers. To counter rapid bot adaptation, Cloudflare is introducing self-adjusting machine learning models under Adaptive Intelligence and non-deterministic defenses such as the AI Labyrinth. These tools allow site owners to poison scraper training data, trap crawlers in synthetic webs, or queue legitimate agents while dynamically penalizing abusive traffic.
Jin-Hee LeeBGP Role model: tracking the adoption of RFC 9234
Border Gateway Protocol route leaks occur when routing announcements propagate beyond their intended customer-provider or peer relationships, causing traffic misdirection and performance degradation. RFC 9234 embeds relationship awareness directly into BGP by establishing negotiated BGP Roles and attaching an Only to Customer (OTC) transitive path attribute to prevent unauthorized route propagation. To assess real-world adoption, Cloudflare monitored OTC attribute transmission across peer networks and discovered that 33.1 percent of IPv4 and 17 percent of IPv6 paths were missing the OTC attribute. Investigation revealed that Tier-1 transit providers AS1299 and AS3257 were stripping the attribute due to legacy defensive BGP error-handling configurations. Following direct engagement, AS1299 updated its configurations to preserve OTC propagation, whereas AS3257 continued stripping it, highlighting the necessity of broader vendor and carrier support across the Internet.
Bryton HerdesIntroducing Radar Researcher: An AI tool for exploring Internet data in plain language
Cloudflare launched the beta of Radar Researcher, an AI tool designed to let users explore global Internet traffic data using plain-language queries. Built entirely on Cloudflare's developer platform, the system uses the Cloudflare Agents SDK within a Cloudflare Worker, maintaining conversation state and SQLite databases inside Durable Objects. The inference layer relies on Workers AI running models like Kimi K2.7 through an ordered fallback chain across three model families to prevent outages, routed via AI Gateway for logging, caching, and guardrails. Radar's frontend renders retrieved data using its existing native interactive charts, while smaller dedicated models generate conversation titles and follow-up suggestions asynchronously. Additionally, Radar introduced support for the WebMCP standard, exposing imperative and declarative tools to allow external browser agents to navigate and interact with the platform directly.
Lai Yi Ohlsen