# Say it once: introducing Bot Preference Sync

[Cloudflare](https://yomu.fyi/company/cloudflare) · Jin-Hee Lee · Aug 21, 2026

**Type:** Announcement

## Summary

Cloudflare announced Bot Preference Sync, a feature across all customer tiers that automatically reflects dashboard AI bot configurations directly in a site's robots.txt file. Website operators previously faced challenges synchronizing static robots.txt directives with edge enforcement rules, which led some crawlers to ignore stated preferences or attempt bypasses. Under this system, Cloudflare generates and prepends directives for Search, Agent, and Training categories to existing robots.txt files using crawler lists tracked in BotBase. Mixed-use crawlers performing both search indexing and training must meet strict transparency criteria, including honoring opt-outs and providing page-level visibility, to avoid edge blocks when training is disallowed. Additionally, onboarding workflows allow ad-supported publishers to automatically set AI model training to Disallow by default while retaining search visibility.

## Context

Maintaining multiple layers of protection on websites is cumbersome when stated robots.txt preferences and edge enforcement rules disagree, which can lead crawlers to disregard preferences or bypass rules. Furthermore, mixed-use crawlers blending search, agent, and training behind a single user agent make it difficult for site owners to separate desired search indexing from unwanted model training.

## Approach / What changed

Cloudflare introduced Bot Preference Sync across all plan tiers to automatically reflect dashboard AI bot configurations (covering Search, Agent, and Training categories) directly into robots.txt. The feature prepends generated directives to existing files and uses BotBase to update crawler lists. Additionally, bot operators performing both search and training must satisfy specific transparency criteria to access content for indexing when Training is disallowed.

## Takeaways

- When site owners enable Bot Preference Sync, generated AI bot directives are prepended to the existing robots.txt file so preexisting Disallow rules remain intact.
- Mixed-use bots performing search and training must satisfy transparency criteria, such as URL-level visibility and AI summary opt-outs, to access content when training is disallowed.
- Ad-monetized publishers can select a dedicated onboarding option that defaults AI training to Disallow while maintaining search indexing.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Privacy](https://yomu.fyi/topic/privacy), [Search](https://yomu.fyi/topic/search)

[Read original post](https://blog.cloudflare.com/bot-preference-sync)