How to Prepare Your Site for the Agentic Browser
A five-step readiness checklist for teams whose sites are increasingly visited by AI agents: expose callable functions, verify bot traffic, measure real-user speed, and control crawler load.
Loading…
Topic
Bot Management
4 posts about Bot Management. Every summary links to the original.
- 4 posts
- 1 company
- 2 explainers
See the idea
Bot Management explainers
Bot Management as an Adaptive Control Loop
Bot management is an architecture, not a rule list: classify traffic, score it with continuously retrained models, enforce with challenges or blocks, then feed the outcomes back into the model.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Continuous accountability for Verified bots
Cloudflare introduces a system to continuously evaluate whether Verified bots honor website preferences, moving beyond identity verification to sustained trust by measuring compliance with robots.txt directives and HTTP 429 rate-limit responses over time. The system looks for repeated patterns of disregard rather than isolated violations, lowering a bot’s trust band if it ignores site preferences or fails to slow down after receiving rate-limit responses, while consistent compliance increases trust. Cloudflare previously de-listed Perplexity’s bot in 2025 for seeking to access opted-out resources and attempting to circumvent controls when blocked, illustrating the need for ongoing accountability. The company plans to share these trust scores publicly on Radar and eventually let customers choose the trust threshold for handling Verified bot traffic on their websites. This forms an accountability loop with Bot Preference Sync and Bot Management, ensuring trust is earned through sustained behavior rather than assumed from identity alone.
Yandex outage: Data shows that instead of going dark, malicious bots went on the move
Reported drone strikes on Yandex data centers from October 7 to 11 caused network traffic to drop 99% below baseline for Yandex Search and 90% for Yandex.Cloud. Using Cloudflare Radar and Bot Management data, the analysis compared pre-event automation baselines with post-strike patterns on both ASNs. The Yandex Search ASN drop consisted almost entirely of verified search crawlers: 93.6% of its bot requests were verified, and 78.1% belonged to YandexBot. On Yandex.Cloud, vulnerability scanning fell 90% and plugin probing dropped 93%, but high-volume operations migrated to other Russian hosting providers, preserving identical JA4 fingerprints. Global bot traffic remained stable at 39.5%, demonstrating that malicious automation moves rather than disappears and that request behavior analysis is necessary to understand threats.
Have it both ways: stay discoverable in search while disallowing AI training
Cloudflare announces a Disallow AI Training setting intended to let website owners refuse AI training while keeping content available to search crawlers. The problem is mixed-use crawlers such as Applebot, Bingbot, and Googlebot, which can serve both search indexing and model training, making a simple block affect discoverability. Cloudflare says its network can publish preferences, identify and classify crawlers, block those that ignore controls, and report operator behavior on Radar; the setting publishes a no-training preference through robots.txt while allowing Accountable mixed-use crawlers for search. Apple, Google, and Microsoft meet or have committed to meet the Accountable requirements, which include training and summary opt-outs, URL-level visibility, metrics, and assurance that training opt-outs will not affect traditional search.
Introducing Adaptive Intelligence: undermining the economics of every bot attack
Cloudflare introduces Adaptive Intelligence, a bot detection engine designed for attackers who distribute requests across proxies, vary fingerprints, and continually test defenses. It argues that deterministic, rule-based systems create stationary targets: attackers receive repeated yes-or-no feedback while defenders update on slower release cycles, making adaptation economically cheaper for attackers. The engine changes this dynamic through continuously retrained machine learning, statistical rather than fixed judgments, and planned disposable rules that appear and retire at random intervals; it also learns from corrections and misses across millions of sites. Its first component, continuous retraining of the machine-learning system behind bot score, is launching now, while disposable rule generation and broader traffic learning are described as forthcoming, with Enterprise customers enabling “Auto Update Machine Learning” to receive it.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.