---
title: "Introducing Supabase Evals"
description: "Supabase has open sourced supabase/evals, a benchmark and framework for testing how AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks such as schema creation, Edge Function debugging, and RLS policy repair, scoring behavior with deterministic checks and LLM-as-a-judge. The framework executes scenarios in hosted-like and local containerized environments through the actual MCP server and CLI, with separate breadth-focused benchmark and depth-focused regression suites, plus a retry after failure. Initial findings show many agents pass broad tasks without skills, while skills improve some scores and documentation use; agents still often choose hand-written migrations, miss newer libraries, and use documentation inconsistently across harnesses."
---

# Introducing Supabase Evals

[Supabase](https://yomu.fyi/company/supabase) · matt\_rossman · Jul 31, 2026

**Type:** Announcement

## Summary

Supabase has open sourced supabase/evals, a benchmark and framework for testing how AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks such as schema creation, Edge Function debugging, and RLS policy repair, scoring behavior with deterministic checks and LLM-as-a-judge. The framework executes scenarios in hosted-like and local containerized environments through the actual MCP server and CLI, with separate breadth-focused benchmark and depth-focused regression suites, plus a retry after failure. Initial findings show many agents pass broad tasks without skills, while skills improve some scores and documentation use; agents still often choose hand-written migrations, miss newer libraries, and use documentation inconsistently across harnesses.

## Context

As more people build Supabase projects through agents rather than by hand, Supabase needed a way to measure agent performance across its CLI, MCP server, agent skills, and documentation. The evaluation also supports identifying recurring failures, verifying fixes, and testing new features without relying on guesswork.

## Approach / What changed

supabase/evals runs real Supabase scenarios in hosted-like and local CLI environments contained in containers. It uses the actual Supabase MCP server and CLI, separates broad benchmark scenarios from deeper regression scenarios, scores results with deterministic checks and LLM-as-a-judge, permits one retry after failure, publishes benchmark results, and refreshes regression results daily.

## Takeaways

- Benchmark scenarios are designed for breadth across product areas, builder journey stages, and cross-product topics, while regression scenarios focus on known failure modes without affecting published benchmark scores.
- Evaluation scoring combines deterministic checks, such as data-access or Edge Function results, with LLM-as-a-judge for semantic judgments; each scenario runs against a real Supabase environment.
- In the Build stage, Opus 5 and Kimi K3 scored 100% without skills, while skills raised Sonnet 5 from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89%.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Monitoring](https://yomu.fyi/topic/monitoring), [Open Source](https://yomu.fyi/topic/open-source), [Testing](https://yomu.fyi/topic/testing)

- Source: [Supabase](https://supabase.com/blog/introducing-supabase-evals)
- Source URL: https://supabase.com/blog/introducing-supabase-evals
- Ingested by Yomu: 2026-08-30T12:47:10.568Z

[Read original post](https://supabase.com/blog/introducing-supabase-evals)