Are Supabase docs agent-friendly? We didn't know, so we checked

Summary

Supabase found that coding agents increasingly use its documentation to configure projects, creating risks when pages omit or bury critical instructions. To measure this, the team built an evaluation for the Row Level Security (RLS) guide, running an agent through realistic tasks and grading outcomes with checklist-based and agent-based judges. The initial runs found unauthenticated anon access with write privileges across tables and showed that test guidance was skipped when separated from the policy code. Supabase responded by adding procedural revocation steps, explicit policy snippets, and colocated test examples; the eval then passed the affected checks. The project concludes that agents and humans both benefit from selective-reading-aware structure, clear sections, explicit signposting, and recurring documentation evaluations, with the RLS scenario moving into the public supabase/evals benchmark.

Takeaways

1. 1

   The first RLS runs granted anon insert, update, delete, and truncate privileges across the todo, list, membership, item, weather-station, and weather-reading tables.

2. 2

   Moving test examples next to the policy code fixed an eval failure caused by agents selectively reading the page instead of following every section.

3. 3

   The RLS scenario is moving from an internal regression suite into the public supabase/evals benchmark, alongside expansion to other popular guides.