Bloom Filters: The Cheap Gate Before the Expensive Question
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
Loading…
Topic
Search
185 posts about Search. Every summary links to the original.
- 185 posts
- 31 companies
- 3 explainers
See the idea
Search explainers
Apache Lucene Scoring: BM25 Relevance vs StableTfl Consistency
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
AI Search: Speed vs. Relevance Trade-off
AI Search balances retrieval speed and result quality by choosing between dense embeddings, sparse lexical search, and hybrid approaches. Understand which trade-off fits your workload.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents
Multimodal search agents typically rely on freeform reasoning before acting, but small models often produce lengthy reasoning that offers little guidance and increases inference cost. The authors propose Selection-Based Structured Reasoning (SSR), which reframes reasoning as selecting from pre-specified natural-language candidates rather than open-ended generation. SSR enables parallel scoring by using teacher-forced prefilling and a shared context KV cache to compute token likelihoods across candidates simultaneously. Evaluated on seven multimodal search benchmarks with 2B and 4B models, SSR maintains task performance under various reinforcement learning objectives and supervised fine-tuning. The framework achieves search success rates competitive with leading agents of the same scale while reducing per-turn reasoning latency by over 90% and total per-question inference latency by 28–54%.
Sizing Lakebase Search: lessons from a real app and 4 million passages
Slide Hub, an internal Databricks app for slide search, uses Lakebase Postgres with Lakebase Search extensions (lakebase_vector and lakebase_text) to run hybrid vector and BM25 search on the same database as application data. The app stores 23k slides and indexes up to 4M passages from the MIRACL English corpus. Benchmarking shows that below 8 Compute Units (CU), Lakebase Postgres is the bottleneck; above 8 CU, the app’s worker count limits throughput. Candidate pool size significantly affects cold-start latency: 200 candidates preserves 90.5% of top-10 results while touching a third of the pages of 1,000 candidates. A 14-minute load test cost 0.13, with Lakebase compute billing only while active. For production, a 0.5 CU minimum costs ~ 31-40/month and avoids cold-start delays.
7 things I wish every search box did - The Intercom Blog
Search in web applications often fails to deliver speed and relevance, yet it is critical for findability as users cannot browse effectively through large amounts of content. The post, by Algolia’s CEO, presents seven principles: show real results from the first keystroke, search across multiple attributes like name and company without field specification, rank based on object importance and context such as user activity or location, highlight matched terms to explain results, offer faceted filters instead of complex syntax, accept typos and synonyms while ranking approximate matches lower, and ensure consistent mobile and desktop experiences. Ranking should consider dimensions like recent activity or geo-location rather than text frequency, making users feel the search reads their minds. Results must explain why they matched to help users quickly identify the best result. These principles help users find results with minimal effort and make search feel intuitive, addressing the iceberg of search complexity.
Enriching Product Catalogs For Agentic Commerce Discovery with MongoDB
AI agents are transforming retail discovery, requiring product catalogs to be machine-readable and context-rich for systematic retrieval, with McKinsey estimating AI agents could orchestrate $3 trillion to $5 trillion globally by 2030. Retailers need a single enriched catalog that projects consistently to three surfaces: outbound product feeds, agentic commerce endpoints via REST APIs or MCP, and product detail pages with schema.org JSON-LD. MongoDB’s document model supports three tiers of enrichment—identity and transactability, filterable attributes, and agentic context such as use cases and semantic embeddings—to help AI agents reason over products. Change streams and aggregation pipelines keep projections current across surfaces, ensuring consistency that AI agents cross-check. MongoDB Search and Vector Search enable hybrid retrieval for intent-based queries, helping retailers improve discoverability and gain a competitive advantage in the agentic commerce era.
Data Modeling and Schema Design for MongoDB Search Part 1: Practical Considerations
Data modeling for MongoDB Search requires balancing conceptual, logical, and physical schema decisions against workload demands and system constraints. The $search stage must be the first in an aggregation pipeline, queries a single Search index, and runs against a single collection, with results returned in relevancy order. The underlying Lucene engine supports two mappings for arrays of sub-documents: default document mapping flattens fields into one index object, while embeddedDocuments mapping creates separate Lucene index objects per sub-document. Embedded mapping enables cross-field matching within sub-documents but explodes the index object count and causes heavy reindexing on any update, as the entire block is deleted and re-added. This post establishes the foundational concepts and physical constraints that will inform schema patterns in Part 2.
AI Search is now generally available
Cloudflare’s AI Search, a managed pipeline combining Workers AI, Vectorize, R2, and Browser Run, is now generally available after a year of preview. The update adds native image embeddings via the Qwen3-VL-Embedding model with Matryoshka Representation Learning, enabling visual search by texture and composition alongside caption-based text search. AI Search now supports files up to 10 MiB and optical character recognition for scanned PDFs; queries undergo optional rewriting, parallel vector and keyword search, fusion, and optional reranking. Previously reliant on caption generation only, image retrieval now embeds query images directly using multimodal models, while text-only models convert images to text via ToMarkdown. Billing starts November 1, 2026, with a free monthly allotment of 5 million ingestion tokens, 10 GB storage, 1,000 semantic queries, and 1,000 full-text queries, with no capacity units or monthly minimums.
Same Query, Same Score: Consistent Ranking with StableTfl in Atlas Search
Paginated search in Atlas Search can show duplicate or skipped results across pages because BM25 scores drift between replicas due to changing corpus statistics. The usual workarounds like sticky routing or index snapshots require extra state and are cumbersome. The solution is StableTfl, a scoring function that uses only term frequency and document length, with term length as a proxy for rarity based on Zipf’s brevity law, eliminating corpus statistics and requiring no extra state. StableTfl is contributed to Apache Lucene and is available as a per-field similarity option in Atlas Search. Benchmarks on BEIR show StableTfl achieves 0.315 NDCG@10 compared to BM25’s 0.346, a 9% relative drop, making it suitable for paginated search, RAG pipelines, and multi-replica deployments where consistency matters more than peak relevance.
Financial Crime Mitigation with MongoDB / Part I: Dynamic Customer Profile
This article is the first in a series on building a financial crime mitigation platform with MongoDB as a unified data platform, focusing on the customer onboarding stage. It outlines a three-step process: capturing prospective customer data into a dynamic profile, running candidate basic checks using hybrid search with the $rankFusion operator, and performing compliance verification via network graphs built with $graphLookup. MongoDB’s flexible data model aggregates structured and unstructured data from various sources, enabling a single customer view. The platform supports vectorized profile representations for similarity searches and clustering, all within one cluster without external search engines. The post concludes by noting that the next part will explore AI-enhanced due diligence processes.
Database for AI Agents: 5 Evaluation Criteria
The post defines a database for AI agents as infrastructure for short-term, episodic, procedural, and operational memory, rather than merely a place to retrieve documents. It proposes five evaluation criteria for moving agent workloads from prototypes to production: per-agent branch isolation, serverless scale-to-zero, hybrid search, ACID guarantees, and unified platform access. The technical rationale is that agents perform repeated reads and writes, resume tasks, share state concurrently, and need current data across vector, keyword, and metadata retrieval; zero-copy branches, sub-second wakeups, and single-query hybrid search address specific operational constraints. It concludes that a database missing more than one minimum bar becomes a production risk, while standard Postgres alone supplies ACID but not branching, scale-to-zero, or unified operational and analytical access.
The Web Search Your Agent Inherited Isn't Good Enough
An agent that keeps a company’s market view current must combine Databricks-hosted internal account records with external web signals such as funding rounds, leadership changes, product launches, and hiring surges. The initial implementation duplicates enrichment logic across Claude Code, Codex, and a direct model API, with inconsistent search results and no shared cost, governance, or audit trail. Omnigent consolidates the agent definition, model, tools, policies, limits, and model-call governance above those harnesses, while letting the engineer select a common search provider. Nimble’s Search API supplies live web data, while its Web Search Agents orchestrate multi-source retrieval and extraction, cross-check sources, return citations, and adapt to difficult sites and recurring retrieval paths. In Nimble’s testing, adding its search increased LLM benchmark accuracy from 46% to 71% and halved web search costs; the proposed Databricks setup then connects internal records and external context under one governance surface.
The guest journey, updated in real time: extending Airbnb’s sequence recommender with Chronon
Airbnb’s guest sequence recommender previously refreshed Transformer-based embeddings in a daily batch, leaving activity from earlier the same day unavailable until the following run and creating up to nearly two days of staleness. To close this gap, Airbnb added Push Mode, which emits notifications when streaming feature values commit, and Near-real-time Model Transform, which invokes the deployed sequence encoder inside that streaming pipeline. Each page view or search is merged with the guest’s long-term and short-term sequence data, triggers inference, and writes a fresh embedding to the same low-latency store used by batch serving. The resulting system typically updates embeddings in under a minute, often within 10–30 seconds, improved offline NDCG by 1.67%, and increased uncancelled bookings by approximately one-third of a percent in online A/B tests.
Building a RAG Pipeline for Semantic Code Search: A Developer Diary and Field Notes
JetBrains Context is presented as a production RAG pipeline for semantic code search, designed to give coding agents precise, citable evidence from repositories instead of relying on keyword tools such as grep. This first installment covers parsing, structure-aware chunking, and vectorization, with the goal of retrieving meaningful functions, symbols, or snippets rather than whole files or insignificant lines. Language parsers produce syntax nodes, and the algorithm retains the largest units that fit, splits oversized leaf nodes when necessary, groups small adjacent units, and keeps documentation, annotations, modifiers, and closing syntax with their constructs. JetBrains Context supports this approach for Kotlin, Java, Python, JavaScript, TypeScript, C#, PHP, Go, and Rust; other languages use line-based splitting. Vectors are generated with an open-weight model on company-operated GPUs, while indexed records store coordinates rather than source code, and the post reports that this setup outperformed hosted embedding APIs on internal code-retrieval benchmarks.
Have it both ways: stay discoverable in search while disallowing AI training
Cloudflare announces a Disallow AI Training setting intended to let website owners refuse AI training while keeping content available to search crawlers. The problem is mixed-use crawlers such as Applebot, Bingbot, and Googlebot, which can serve both search indexing and model training, making a simple block affect discoverability. Cloudflare says its network can publish preferences, identify and classify crawlers, block those that ignore controls, and report operator behavior on Radar; the setting publishes a no-training preference through robots.txt while allowing Accountable mixed-use crawlers for search. Apple, Google, and Microsoft meet or have committed to meet the Accountable requirements, which include training and summary opt-outs, URL-level visibility, metrics, and assurance that training opt-outs will not affect traditional search.
Evolving Pinterest’s Embedding Retrieval Platform
Pinterest describes how its Manas distributed search platform is evolving embedding retrieval for a corpus growing toward tens of billions of embeddings. Traditional HNSW indexes require all data in RAM, while single-vector two-tower retrieval limits context-dependent relevance, motivating changes across quantization, SSD-based serving, and multi-embedding retrieval. Manas uses Scalar Quantization and Product Quantization across HNSW and IVF, with SQ preserving over 90% recall while reducing index size, and Linear Scaling plus SIMD cuts per-query computing resources by 10–15%. For disk-based serving, SPANN with PQ is reported as 4.5x faster than plain SPANN and over 3x DiskANN QPS, while a five-billion-embedding production evaluation saved over 40% CPU time with under 5% recall drop. The platform is also adding ColBERT-style late interaction using Sum of MaxSim, with a pilot in progress.
Debugging our AI search assistant with agent tracing
Sentry’s Search Query Assistant converts natural-language prompts into Sentry Search Syntax, but its non-deterministic output requires evals and debugging when generated queries are wrong. A production issue caused searches on custom numerical attributes to return no events, so the team used Sentry’s AI Conversation view and Agents filters to find the relevant trace, inspect the timeline, and follow tool calls into the backend API. The trace showed that tool results were being incorporated into the system prompt with the field name bug__predictions_count instead of the required tags[bug_predictions_count, number] form. The team added an eval scenario, fixed the prompt generation, verified the corrected query locally and in production, and confirmed the updated prompt and API parameter in the trace. They also instrumented local eval runs to send data to Sentry, allowing debugging before a pull request.
Halving the Time: How Uber Eats Rebuilt Its Search Pipeline
Uber Eats rebuilt its search pipeline after accumulated architectural decisions made latency increasingly costly across retrieval, hydration, ranking, ads, and presentation. The team changed its primary metric from backend API response time to Above-the-Fold completion, then introduced paginated responses with server-side caching and concurrent template rendering; these changes improved ATF by over 200 milliseconds. Other work removed low-yield retrieval strategies, grouped items by product identity to reduce lookups by over 100 times, split ranking and presentation hydration, eliminated false DAG dependencies, added request hedging, and redesigned ad data as column-oriented in memory. The combined effort cut search latency in half, including approximately 120 milliseconds from retrieval, 100-plus from hydration changes, and 130 from ads, while planned product-based search, zero-pass ranking, and streaming target further gains.
Adaptive Instructed-Retriever: Frontier-Quality Search at 2x Lower Latency
Adaptive Instructed-Retriever addresses the tension between retrieval quality and latency for enterprise data agents, especially on multi-hop questions that benefit from sequential search. It combines parallel single-step retrieval with bounded multi-step search, using a fixed upper limit and an adaptive stopping policy that spends extra steps only when they are likely to improve results. The model is trained with online reinforcement learning using CISPO, a reward that balances trajectory quality against search cost, and synthetic multi-hop data alongside Instructed-Retriever-1 data. Across seven held-out internal and external benchmarks, it matches leading third-party and open-source models while answering in 5.8 seconds, reported as more than twice as fast as the compared alternatives, and substantially improves over single-step search. Varying the step penalty produces checkpoints spanning different quality-latency tradeoffs for production workloads.
NeoMME: an efficient Multimodal-native and Multilingual Encoder
NeoMME is a family of 260M and 800M multilingual multimodal encoders designed to represent text and images without a separate pretrained vision tower or causal language model. A single bidirectional Transformer processes text tokens and raw 32×32 image patches, while pretraining from scratch uses a masked discrete-diffusion text-denoising objective with high masking to encourage image-grounded reconstruction. The retrieval variant adds jointly trained dense mean-pooled and 128-dimensional late-interaction heads, returning both in one forward pass for page-image retrieval. On ViDoRe v3, the 260M model reaches 0.523 and both sizes lie on the model-size Pareto frontier; at 2048×2048, it encodes about 51 pages per second on an NVIDIA L40S, roughly twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction storage from roughly 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10.
Give Your Coding Agents a Memory You Own
funes is presented as a durable memory layer for coding agents that turns existing session traces into searchable, provenance-backed working memory across machines and agent types. A single binary adds recall and get tools to Claude Code, Codex, pi, or Hermes, builds an incremental local index, and automatically indexes completed turns. Its deterministic pipeline normalizes traces, chunks them, embeds them with a pinned local model, stores them in Lance, and combines vector and BM25 search with rank fusion, cross-encoder reranking, recency weighting, and neighboring chunks. Memories can remain local or be bound to a private Hugging Face dataset owned by the user, with credential redaction, secret scanning, local caching, and Hub-provided access control and versioning. In the handoff-vs-recall benchmark, recall was the cheapest option on both tasks, costing 8x less than a written handoff on one and 4x less on the other, while compaction succeeded on only one task.
BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agents
Cloudflare launched BotBase for Operators to give bot operators a clearer way to submit, monitor, update, and cancel bot-directory submissions from the dashboard. The new area includes a bots directory, submission form, and history tab with Waiting for review, Accepted, and Rejected statuses, including rejection reasons and classification changes. Its intake model records what a bot does, how it uses content, and whether it is direct or intermediary, while allowing multiple behaviors and Content Signals declarations. Reviews now use automated checks for duplicates, user-agent specificity, and verification through IP lists, reverse DNS, or Web Bot Auth, routing exceptions to staff. The launch focuses on visibility, with ownership, observability, and conversation identified as future goals.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.