---
title: "Have it both ways: stay discoverable in search while disallowing AI training"
description: "Cloudflare announces a Disallow AI Training setting intended to let website owners refuse AI training while keeping content available to search crawlers. The problem is mixed-use crawlers such as Applebot, Bingbot, and Googlebot, which can serve both search indexing and model training, making a simple block affect discoverability. Cloudflare says its network can publish preferences, identify and classify crawlers, block those that ignore controls, and report operator behavior on Radar; the setting publishes a no-training preference through robots.txt while allowing Accountable mixed-use crawlers for search. Apple, Google, and Microsoft meet or have committed to meet the Accountable requirements, which include training and summary opt-outs, URL-level visibility, metrics, and assurance that training opt-outs will not affect traditional search."
---

# Have it both ways: stay discoverable in search while disallowing AI training

[Cloudflare](https://yomu.fyi/company/cloudflare) · Bryan Becker · Sep 15, 2026

**Type:** Announcement

## Summary

Cloudflare announces a Disallow AI Training setting intended to let website owners refuse AI training while keeping content available to search crawlers. The problem is mixed-use crawlers such as Applebot, Bingbot, and Googlebot, which can serve both search indexing and model training, making a simple block affect discoverability. Cloudflare says its network can publish preferences, identify and classify crawlers, block those that ignore controls, and report operator behavior on Radar; the setting publishes a no-training preference through robots.txt while allowing Accountable mixed-use crawlers for search. Apple, Google, and Microsoft meet or have committed to meet the Accountable requirements, which include training and summary opt-outs, URL-level visibility, metrics, and assurance that training opt-outs will not affect traditional search.

## Context

The announcement addresses the tradeoff between allowing search indexing and allowing AI training when a single crawler performs both functions. It also describes Cloudflare’s Accountable designation for operators that provide control and transparency over training and AI summaries.

## Approach / What changed

Cloudflare adds a domain-level Disallow AI Training control that publishes the applicable no-training preference in robots.txt. Accountable mixed-use crawlers remain allowed for search, while other training crawlers are blocked; Cloudflare also identifies crawler behavior, reports operator actions on Radar, and applies existing Block settings to mixed-use crawlers.

## Takeaways

- Disallow AI Training allows Accountable mixed-use crawlers to continue search crawling while blocking other training crawlers, including training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI.
- Block and Block on pages with ads now apply to mixed-use crawlers, so they can affect search visibility as well as AI training. Owners that want those crawlers removed entirely must select Block.
- Cloudflare classifies crawler behavior as Search, Training, or Agent. There is currently no Disallow setting for Agents because the Internet lacks a well-established directive for expressing that preference.

## FAQ

### How does Disallow AI Training differ from Block?

Disallow AI Training keeps Accountable mixed-use crawlers allowed for search while blocking other training crawlers. Block stops all crawlers, including mixed-use crawlers, so it affects search discoverability.

### Why can’t robots.txt alone control AI training?

A robots.txt directive cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it. Cloudflare says its network can publish preferences, classify crawlers, block noncompliant ones, and report operator behavior.

### Does Cloudflare provide a Disallow setting for agent crawlers?

Not currently. Agents do not create the same search-discoverability tradeoff, and there is not yet a well-established directive for expressing Disallow preferences to them; Cloudflare says it will revisit the approach as standards such as ai-prefs mature.

**Tags:** [AI](https://yomu.fyi/topic/ai), [Bot Management](https://yomu.fyi/topic/bot-management), [Cloudflare](https://yomu.fyi/topic/cloudflare), [Search](https://yomu.fyi/topic/search)

- Source: [Cloudflare](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers)
- Source URL: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers
- Ingested by Yomu: 2026-09-15T16:01:42.067Z

[Read original post](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers)