How to Prepare Your Site for the Agentic Browser
A five-step readiness checklist for teams whose sites are increasingly visited by AI agents: expose callable functions, verify bot traffic, measure real-user speed, and control crawler load.
CloudflareNic JansmaAnnouncement
Cloudflare has released the BEACON dataset, an anonymized collection of billions of real-user performance measurements from 10,000 of the largest websites. The dataset reports Core Web Vitals (LCP, CLS, INP) with full histograms, enabling analysis of any percentile including the long tail. It reveals that WebKit browsers underperform Blink in 46 countries, and that resource-discovery and render-blocking contribute more to slow LCP than download time. Soft navigations in single-page applications load two to three times faster than hard navigations, but landing pages remain heavy. The dataset is publicly available on Google BigQuery and includes differential privacy via the RAPPOR algorithm to protect individual user data.
The post discusses the gap between developer perception and real-world web performance, noting that developers often test on powerful hardware while end users face slower devices, limited data plans, or poor infrastructure. Cloudflare’s BEACON dataset aims to close that gap by providing billions of anonymized real-user measurements from 10,000 large websites, updated daily in Google BigQuery.
Cloudflare built the BEACON dataset by collecting Real User Monitoring data from 10,000 of the largest websites on its network, normalizing records to a uniform beacon count, applying differential privacy via the RAPPOR algorithm, aggregating by common dimensions (country, OS, browser, connection), and discarding groups with fewer than five data points. The data is published as histograms in Google BigQuery.
WebKit browsers show worse LCP and INP than Blink-based browsers in 46 countries where they account for over 10% of traffic, with LCP up to 50% worse in Cambodia.
For pages with poor LCP, the largest delays come from discovering the LCP resource (load time) and render blocking, not from downloading the resource itself.
Soft navigations in SPAs are two to three times faster than hard navigations at every percentile, but landing pages for SPAs can have median LCP of 1.37 seconds and P95 of 8.94 seconds.
BEACON uses the RAPPOR algorithm for differential privacy, limits data collection to the top 10,000 websites, and aggregates records with at least five data points to prevent identifying individuals or specific sites.
It reports the three Core Web Vitals: Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS), and Interaction to Next Paint (INP), along with sub-parts for LCP and INP that break down loading and interaction stages.
The dataset is publicly available on Google BigQuery, with example queries provided in the post. Users can join BEACON with other datasets like World Bank GDP per capita for additional analysis.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A five-step readiness checklist for teams whose sites are increasingly visited by AI agents: expose callable functions, verify bot traffic, measure real-user speed, and control crawler load.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.