How to Prepare Your Site for the Agentic Browser
A five-step readiness checklist for teams whose sites are increasingly visited by AI agents: expose callable functions, verify bot traffic, measure real-user speed, and control crawler load.
CloudflareMichelle ZatlynProblem & solution
Cloudflare’s 2026 founders’ letter, marking the company’s 16th birthday, examines rapid changes in internet traffic driven by AI. Since mid-2025, new website creation has exploded, fueled by ‘vibe coding’ tools that allow non-programmers to build apps, with over 7 million developers now using Cloudflare’s platform. Automated traffic from AI agents and crawlers is projected to surpass human traffic by May 2026 and could reach 1,000 times human levels within five years. This surge risks a tragedy of the commons: agents consume resources from many sites but benefit only one, threatening small businesses that lack the information volume older competitors hold. Cloudflare proposes reducing crawler load by fetching only changed content and enabling creators to receive payment when agents use their content, alongside partnerships to shape a fairer internet.
Annual founders’ letter reflecting on Cloudflare’s 16th anniversary, discussing AI’s impact on internet growth, automated traffic, and risks to small businesses, along with proposed solutions.
Cloudflare is working on a solution that allows crawlers to fetch only new or changed content, reducing the load on websites, and providing a mechanism for content creators to receive payment when AI agents use their content. The company will also announce partnerships with other organizations that share the goal of a fair and sustainable internet.
AI-powered ‘vibe coding’ tools have enabled a surge in new website creation since mid-2025, with over 7 million developers building on Cloudflare’s platform.
Automated traffic from AI agents is expected to exceed human traffic by May 2026 and could reach 1,000 times human levels in five years.
AI agents threaten small businesses by disproportionately benefiting from their content without compensation, leading to market consolidation; Cloudflare proposes reducing crawler load and paying creators.
The rise of AI agents and crawlers, which read many pages to answer queries, is driving the growth. Cloudflare originally forecast this would happen in 2027, but it's now expected by May 2026, and could be 1,000 times human traffic in five years.
Agents survey many sites but only recommend one, so the other 999 bear the load without benefit. They favor well-known sites, making it harder for new entrants to gain visibility, leading to consolidation.
Cloudflare is working so crawlers fetch only changed content, reducing site load. It also offers a way for content creators to get paid when agents use their content, and will announce partnerships for a fairer internet.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A five-step readiness checklist for teams whose sites are increasingly visited by AI agents: expose callable functions, verify bot traffic, measure real-user speed, and control crawler load.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.