Topic

Crawlers

1 post about Crawlers. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Cloudflare
Cloudflare and OpenAI: making AI discovery better for everyone
Cloudflare and OpenAI launched a research pilot in July 2026 to help ChatGPT users discover content more efficiently using network signals about page existence changes and potential harm. The pilot ran on a cohort of about one percent of Cloudflare’s network and has enabled OpenAI to index hundreds of millions of new web pages that are useful but sparsely linked. Cloudflare provides these discovery signals to OpenAI’s crawler while OpenAI independently evaluates and ranks pages by relevance and helpfulness for ChatGPT Search. The collaboration also tests signals that flag harmful pages like fake login forms so crawlers avoid them without penalizing the legitimate domain. The next phase will refine page selection test faster change signals and measure business outcomes for site owners whose content is discovered.

The best engineering reads, once a week.

Topic

1 post about Crawlers. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Cloudflare
Cloudflare and OpenAI: making AI discovery better for everyone
Cloudflare and OpenAI launched a research pilot in July 2026 to help ChatGPT users discover content more efficiently using network signals about page existence changes and potential harm. The pilot ran on a cohort of about one percent of Cloudflare’s network and has enabled OpenAI to index hundreds of millions of new web pages that are useful but sparsely linked. Cloudflare provides these discovery signals to OpenAI’s crawler while OpenAI independently evaluates and ranks pages by relevance and helpfulness for ChatGPT Search. The collaboration also tests signals that flag harmful pages like fake login forms so crawlers avoid them without penalizing the legitimate domain. The next phase will refine page selection test faster change signals and measure business outcomes for site owners whose content is discovered.

The best engineering reads, once a week.