---
title: "Apache Lucene Scoring: BM25 Relevance vs StableTfl Consistency"
description: "A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages."
---

# Apache Lucene Scoring: BM25 Relevance vs StableTfl Consistency

**Type:** [Comparison](https://yomu.fyi/visual-explainers/category/comparison.md)

![A two-column comparison diagram of Apache Lucene scoring: BM25 on the left and StableTfl on the right, showing relevance, consistency, pagination, and a final decision cue.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/2338ba81-6b7e-4490-9bd5-b5808d46b286/image-a76665c9-4ab0-4f8f-90cc-03109ceb0529.webp)

> A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.

## Explanation

Apache Lucene scores documents with BM25 using corpus-wide statistics such as document frequency, which makes relevance strong but causes scores to drift as replicas and indexes change. StableTfl is a Lucene similarity that drops corpus statistics and scores using term frequency, document length, and term length as a rarity proxy. In a comparison, BM25 wins on peak relevance, while StableTfl wins on repeatable pagination and multi-replica consistency. The right choice depends on whether your search is ranked once for discovery or must return the same order across pages, replicas, and RAG retrieval steps.

## FAQ

### Why do BM25 scores drift across pages or replicas?

BM25 uses corpus-wide statistics such as document frequency. As the corpus changes or replicas see different index states, the same query can receive different scores, so pagination can duplicate or skip results. Workarounds like sticky routing or index snapshots add extra state to manage.

### What do you give up by switching from BM25 to StableTfl?

StableTfl drops corpus statistics and scores with term frequency, document length, and term length as a rarity proxy. In BEIR benchmarks it reaches 0.315 NDCG@10 versus BM25's 0.346, about a 9% relative drop. Choose it when stable ordering matters more than peak relevance.

### When should I keep BM25 in a Lucene-based search stack?

Keep BM25 when peak relevance is the priority and you can absorb consistency work with extra state. StableTfl is better suited to paginated search, RAG pipelines, and multi-replica deployments where the same query should return the same score.

**Tags:** [Apache Lucene](https://yomu.fyi/topic/apache-lucene.md), [Benchmark](https://yomu.fyi/topic/benchmark.md), [BM25](https://yomu.fyi/topic/bm25.md), [Latency](https://yomu.fyi/topic/latency.md), [RAG](https://yomu.fyi/topic/rag.md), [Search](https://yomu.fyi/topic/search.md)

## Related reading

### [What is AI agent orchestration? How it works in 2026](https://yomu.fyi/post/what-is-ai-agent-orchestration-how-it-works-in-2026.md)
- Company: [Twilio](https://yomu.fyi/company/twilio.md)
- Author: Jesse Sumrak, Luke Morgan
- Published: Sep 25, 2026
- Type: Explainer

AI agent orchestration coordinates multiple AI agents working on a single task, deciding routing, carrying context, resolving contradictions, and escalating to humans when needed. It operates at two levels: the agent level, where frameworks like LangGraph route between specialized agents, and the conversational AI level, which preserves customer identity and history across Voice, SMS, and WhatsApp channels. Orchestration comprises four jobs: routing, shared memory and state, conflict resolution, and human escalation, with parallel execution requiring robust memory management. Twilio’s Conversation Orchestrator provides a shared conversation record as a system of record, while Conversation Memory extracts observations and resolves conflicts by updating records with the most recent version. The FAQ section addresses differences from workflow automation, comparison criteria for orchestration tools, and the mechanism for agent-to-human handoffs within the same session without callbacks.


### [Same Query, Same Score: Consistent Ranking with StableTfl in Atlas Search](https://yomu.fyi/post/same-query-same-score-consistent-ranking-with-stabletfl-in-atlas-searc.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Published: Sep 25, 2026
- Type: Explainer

Paginated search in Atlas Search can show duplicate or skipped results across pages because BM25 scores drift between replicas due to changing corpus statistics. The usual workarounds like sticky routing or index snapshots require extra state and are cumbersome. The solution is StableTfl, a scoring function that uses only term frequency and document length, with term length as a proxy for rarity based on Zipf’s brevity law, eliminating corpus statistics and requiring no extra state. StableTfl is contributed to Apache Lucene and is available as a per-field similarity option in Atlas Search. Benchmarks on BEIR show StableTfl achieves 0.315 NDCG@10 compared to BM25’s 0.346, a 9% relative drop, making it suitable for paginated search, RAG pipelines, and multi-replica deployments where consistency matters more than peak relevance.


### [Turn your REST APIs into MCP tools with Google Cloud API Gateway](https://yomu.fyi/post/turn-your-rest-apis-into-mcp-tools-with-google-cloud-api-gateway.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Sanjay Pujare, Paul Howell
- Published: Sep 24, 2026
- Type: Announcement

Most enterprise REST APIs are invisible to AI agents, forcing teams to run separate MCP servers that duplicate their existing gateway logic. Google Cloud API Gateway in Public Preview solves this by acting as a remote MCP server — annotate the OpenAPI 3.0.x/3.1.x spec with x-google-api-management.mcp and deploy it with no extra infrastructure. The gateway accepts MCP JSON-RPC requests on a single /mcp endpoint, transcodes tools/call into the corresponding REST call, runs it through existing policies, and returns the response as an MCP result. Authentication via JWT or API key works unchanged for REST operations; tools/list is unauthenticated by default and requires JWT in production for security. With no separate server to operate, teams in Public Preview can expose up to 1,000 tools per gateway that support existing authentication and quota, and the MCP server is automatically discoverable via API hub and Agent Registry.


### [Unity Catalog Pages: a governed home for your business knowledge in Genie Ontology](https://yomu.fyi/post/unity-catalog-pages-a-governed-home-for-your-business-knowledge-in-gen.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Tim Feng, Ashish Singh
- Published: Sep 23, 2026
- Type: Announcement

Business concepts like active customer or completed trip are often defined differently across teams, scattered in Slack threads and spreadsheets, causing AI agents to guess confidently when answering questions. Databricks introduces Unity Catalog Pages as a governed home within Genie Ontology where data stewards with Genie Code define the authoritative meaning of those terms and entities. Pages complement existing semantics — metric views how to calculate, domains where to look, and certification what to trust — by capturing what each concept means. Genie Code can bootstrap Pages by extracting definitions from Confluence, Slack, Google Docs, or GitHub via MCP-connected tools, turning weeks of work into minutes. When Genie One uses a Page to ground an answer, the Page is cited as a clickable source so users can inspect the definition and verify the reasoning.


### [How I massively improved my AI inference performance without buying new hardware](https://yomu.fyi/post/how-i-massively-improved-my-ai-inference-performance-without-buying-ne.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Boaz Ben Shabat
- Published: Sep 23, 2026
- Type: Problem & solution

Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.