---
title: "Crawlers"
description: "1 posts about Crawlers, summarised, each linking to the original."
---

# Crawlers
> 1 posts about Crawlers, summarised, each linking to the original.

## Articles

### [Cloudflare and OpenAI: making AI discovery better for everyone](https://yomu.fyi/post/cloudflare-and-openai-making-ai-discovery-better-for-everyone.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: James Balfour (OpenAI)
- Published: Oct 15, 2026
- Type: Announcement

Cloudflare and OpenAI launched a research pilot in July 2026 to help ChatGPT users discover content more efficiently using network signals about page existence changes and potential harm. The pilot ran on a cohort of about one percent of Cloudflare’s network and has enabled OpenAI to index hundreds of millions of new web pages that are useful but sparsely linked. Cloudflare provides these discovery signals to OpenAI’s crawler while OpenAI independently evaluates and ranks pages by relevance and helpfulness for ChatGPT Search. The collaboration also tests signals that flag harmful pages like fake login forms so crawlers avoid them without penalizing the legitimate domain. The next phase will refine page selection test faster change signals and measure business outcomes for site owners whose content is discovered.