---
title: "The Internet has a second audience"
description: "More than half of Internet traffic is now automated, and AI agents are the fastest-growing part of it, with daily requests on Cloudflare’s network growing by more than 1,700% over the past year. This second audience consumes bandwidth without generating ad impressions or subscriptions, causing revenue per request to fall while costs rise. Cloudflare is giving site owners tools to see who is visiting, set terms per bot type, and charge for access. These include AI Crawl Control, Web Bot Auth for cryptographically verified agents, Disallow AI Training to block training without losing search indexing, and Pay Per Use to pay publishers when content is actually used. The goal is to build open standards so any site can set its terms and get paid, rather than a few companies controlling discovery and payment."
---

# The Internet has a second audience

[Cloudflare](https://yomu.fyi/company/cloudflare) · Matthew Conroy · Sep 30, 2026

**Type:** Announcement

## Summary

More than half of Internet traffic is now automated, and AI agents are the fastest-growing part of it, with daily requests on Cloudflare’s network growing by more than 1,700% over the past year. This second audience consumes bandwidth without generating ad impressions or subscriptions, causing revenue per request to fall while costs rise. Cloudflare is giving site owners tools to see who is visiting, set terms per bot type, and charge for access. These include AI Crawl Control, Web Bot Auth for cryptographically verified agents, Disallow AI Training to block training without losing search indexing, and Pay Per Use to pay publishers when content is actually used. The goal is to build open standards so any site can set its terms and get paid, rather than a few companies controlling discovery and payment.

## Context

For decades the web was funded by human visitors who saw ads or bought subscriptions. Now more than half of Internet traffic is automated, and AI agents — software fetching pages on a person’s behalf — are the fastest-growing segment. Revenue per request is falling while costs rise, because agents consume bandwidth without generating ad impressions or subscriptions. The old arrangement where being found and getting paid were the same thing has broken down.

## Approach / What changed

Cloudflare is building a set of tools and standards — AI Crawl Control, Business Insights, BotBase, Web Bot Auth, Disallow AI Training, Pay Per Use, Markdown for Agents, and WebMCP — that give site owners visibility into automated traffic, controls to set terms per bot type, and a payment mechanism for content used by agents.

## Takeaways

- AI agents now drive the fastest-growing segment of automated traffic, with daily requests on Cloudflare’s network up more than 1,700% in one year.
- Revenue per request is falling because agents consume bandwidth without generating ad impressions or subscriptions, while costs for bandwidth and compute rise.
- Cloudflare’s Pay Per Use pays publishers when content is actually used by a verified buyer, not for the crawl, and provides usage reports showing what was used and when.

## FAQ

### How can a site verify a real AI agent?

Cloudflare's Web Bot Auth lets operators like OpenAI, Google, and AWS cryptographically sign their agents' requests. This allows a site to verify a real agent from an impersonator without relying on IP addresses or user-agent strings. Cloudflare sees more than 500 billion verified bot requests each week.

### How does Pay Per Use work?

Pay Per Use does not charge for the crawl. It pays when content is actually used. A verified buyer reports each use, Cloudflare checks those reports, then bills the buyer and pays the publisher. Publishers see what was used, when, and what they earned.

### Can a site block AI training without losing search traffic?

Cloudflare's Disallow AI Training keeps a site indexed for search while instructing the operator not to use data for training. Apple, Google, and Microsoft have committed to honor it. This allows a site to be found without being exploited for model building.

**Tags:** [AI](https://yomu.fyi/topic/ai), [AI Agents](https://yomu.fyi/topic/ai-agents), [API](https://yomu.fyi/topic/api), [Cloudflare](https://yomu.fyi/topic/cloudflare), [Security](https://yomu.fyi/topic/security)

- Source: [Cloudflare](https://blog.cloudflare.com/agentic-web)
- Source URL: https://blog.cloudflare.com/agentic-web
- Ingested by Yomu: 2026-09-30T15:00:07.511Z

[Read original post](https://blog.cloudflare.com/agentic-web)