Loading…
Topic
2 posts about BM25. Every summary links to the original.
See the idea
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Traditional OLTP systems like Postgres face challenges meeting the low-latency, high-accuracy search demands of AI agents, often requiring standalone search engines with ETL pipelines. Lakebase introduces lakebase_vector and lakebase_text, two extensions that bring scalable approximate nearest neighbor search and BM25 full-text search directly into Lakebase Postgres. lakebase_vector overcomes pgvector’s memory-bound HNSW index by using hierarchical IVF clustering and binary quantization, achieving 71-millisecond P99 latency at 97 percent recall on the 100-million-vector LAION benchmark. The extensions decouple storage from compute for pay-per-use scaling, enable fast parallel index builds, and support native hybrid search that fuses vector similarity with keyword relevance in a single SQL query. Early adopters like Conexiom reduced infrastructure costs by three times while achieving five times higher throughput, and the service is generally available on AWS and Azure.
Paginated search in Atlas Search can show duplicate or skipped results across pages because BM25 scores drift between replicas due to changing corpus statistics. The usual workarounds like sticky routing or index snapshots require extra state and are cumbersome. The solution is StableTfl, a scoring function that uses only term frequency and document length, with term length as a proxy for rarity based on Zipf’s brevity law, eliminating corpus statistics and requiring no extra state. StableTfl is contributed to Apache Lucene and is available as a per-field similarity option in Atlas Search. Benchmarks on BEIR show StableTfl achieves 0.315 NDCG@10 compared to BM25’s 0.346, a 9% relative drop, making it suitable for paginated search, RAG pipelines, and multi-replica deployments where consistency matters more than peak relevance.
The best engineering reads, once a week.
