Bloom Filters: The Cheap Gate Before the Expensive Question
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
Loading…
Topic
Search
181 posts about Search. Every summary links to the original.
- 181 posts
- 31 companies
- 3 explainers
See the idea
Search explainers
Apache Lucene Scoring: BM25 Relevance vs StableTfl Consistency
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
AI Search: Speed vs. Relevance Trade-off
AI Search balances retrieval speed and result quality by choosing between dense embeddings, sparse lexical search, and hybrid approaches. Understand which trade-off fits your workload.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
What a difference 10 years of open source makes
ClickHouse marks ten years since becoming open source under the Apache 2.0 license, crediting more than 2,600 contributors and a decade of monthly releases for its growth into one of the world’s most widely deployed analytical databases. Recent engineering work spans joins, updates and deletes, JSON, full-text search, vector search, query acceleration, integrations, and developer tooling. Join-focused changes made the TPC-H SF100 workload 26× faster between versions 22.4 and 26.4, while statistics-based reordering reduced one six-table query from 3,903.7 seconds to 2.7 seconds. The project also publishes open benchmarks, including ClickBench, JSONBench, CostBench, and PostgresBench, with scripts, configurations, data, and results intended for independent validation. Its next phase continues to emphasize community contributions and broader ways to use ClickHouse.
What is enterprise intelligence?
Enterprise intelligence (EI) is presented as an organization-wide capability combining business intelligence, knowledge management, enterprise search and AI to turn structured and unstructured data into decisions and actions. Unlike traditional BI, which centers on dashboards, reports and structured data, EI connects these capabilities through a shared architecture and governed business context. The described stack includes a lakehouse-based data foundation, batch and streaming pipelines, governance, semantics, analytics, search, machine learning, generative AI and a decision layer. Shared definitions such as “active customer” and “monthly revenue” are intended to keep dashboards, queries and AI agents aligned, while maintained context addresses knowledge that becomes stale as the business changes. The result described is a common trusted source from which people, applications and agents can produce insights and initiate actions.
How Dropbox uses MCP and Dash to close the design-to-code security gap
Dropbox developed a system using the Model Context Protocol (MCP) and Dash's semantic search to bridge the gap between security threat models and code implementation. By retrieving original security documents during pull requests, an LLM agent automatically evaluates whether the proposed code adheres to previously agreed-upon security requirements. This approach surfaces design regressions and missing controls that traditional static analysis tools miss.
Production-Ready Agents Need A Production-Ready Data Platform
AI development teams face constant shifts in model providers and agent frameworks, demanding data platforms that provide scalable, real-time context management. Agentic workloads require blending unstructured enterprise data, short-term session state, and persistent long-term memory. MongoDB addresses these requirements through its native JSON document model and integrated retrieval capabilities, combining full-text search, vector search, and hybrid search directly over operational data. Customer implementations such as DevRev, ElevenLabs, and Adobe rely on Atlas to achieve sub-100 millisecond hybrid retrieval and handle billions of requests. Additionally, MongoDB is collaborating with LangChain and ecosystem partners to establish open reference architectures and shared interfaces for portable agent memory across frameworks.
Datadog and ClickHouse partner to bring full-fidelity data to modern observability
ClickHouse and Datadog announce a partnership aimed at reducing the tradeoff between retaining complete telemetry and keeping it accessible through familiar observability workflows. New Preview capabilities let organizations route logs directly to ClickHouse through Datadog Observability Pipelines and search those logs from the Datadog Log Explorer. Pipelines can parse, enrich, filter, transform, and redact data before storage, while ClickHouse Cloud’s separation of storage and compute supports long-term retention, independently scaled query resources, and workload isolation. Federated log search queries data directly in ClickHouse, avoiding duplicate copies or re-ingestion while allowing high-volume logs to remain searchable and historical investigations to use full-fidelity telemetry. The partnership combines Datadog’s collection and investigation experience with ClickHouse’s storage and analytics foundation; the native integration and federated search are available in Preview.
What is AI Search?
AI search interprets natural-language questions using semantic understanding and large language models, then returns synthesized answers grounded in cited sources. Unlike traditional search, which matches query words to indexed pages, it tracks context and intent for conversational follow-ups. Its pipeline includes query understanding, embeddings, vector search, retrieval, LLM synthesis, and citations; retrieval-augmented generation (RAG) connects generated responses to source material. The article distinguishes consumer tools that search the open web from enterprise systems that retrieve private, governed company data while respecting access permissions. It concludes that AI search can improve direct information access but should not be treated as definitive, and that sensitive business use requires purpose-built platforms with governance, security, and retrieval quality.
3x Faster Search: Parallel Test-Time Scaling with Instructed-Retriever-1
Instructed-Retriever-1 is introduced as the retrieval-specialized model behind an update to Agent Bricks Knowledge Assistant, targeting search quality and latency. It parallelizes query and filter generation for broader candidate recall, then applies multi-pivot groupwise reranking to improve precision, using one model for both retrieval stages. The model was trained on synthetic enterprise-style environments and evaluated on realistic Knowledge Assistant workloads, where it scored 81.0 nDCG@10 for reranking and matched Claude Sonnet 4.5 retrieval quality on KARLBench. Serving uses a Mixture-of-Experts architecture, FP8 quantization, speculative decoding, and infrastructure tuning; the update reports more than 3x faster search, 2x faster answer generation, and TTFT around two seconds. End-to-end latency was consistently below 10 seconds in the offline evaluation setup.
Agentic Supplier Management with MongoDB Atlas, Voyage AI, and Multi-Modal Search
Retailers face major operational bottlenecks when managing supply chain disruptions through legacy enterprise resource planning systems and siloed documents. Traditional tabular structures struggle to index multi-modal data like images and PDF contracts, delaying critical supplier discovery during regional crises or tariff changes. To overcome these constraints, organizations decouple supplier management into a dedicated operational data layer using MongoDB Atlas and Voyage AI. This architecture stores polymorphic supplier profiles and high-dimensional vector embeddings within a unified collection, enabling natural language semantic search for alternate partners. Real-time updates propagate across systems using MongoDB Change Streams, allowing businesses to rapidly assess supply chain impacts and identify alternative vendors in minutes.
From decentralized Docs-as-Code to a centralized repository: Evolving Grab's documentation strategy
Grab adopted Docs-as-Code in early 2021 to bring technical documentation into engineers’ code workflows, addressing scattered sources, version confusion, discoverability, and quality gaps. Documentation was written in Markdown, stored in version-controlled repositories, reviewed through merge requests, checked in CI/CD, and surfaced through an internal developer portal. As engineering scaled, decentralized repositories produced inconsistent structures and terminology, unreliable organization-wide quality automation, aging content, and tracking problems when source locations changed. Grab therefore moved to a centralized repository while retaining Docs-as-Code, adding shared templates and linters, unified search through fuzzy title lookup and Glean, automated synchronization during migration, and training and feedback programs. The post concludes that decentralized, centralized, or hybrid models fit different contexts, with centralization emphasizing discoverability, governance, consistency, ownership metadata, and analytics.
Beyond the Menu Tree: How Yelp Built a Smarter Customer Success Chatbot with AI
Yelp transitioned from a legacy customer support chatbot that relied on rigid phrase matching and static menus to a retrieval-augmented generation system. The updated architecture classifies incoming queries using a large language model to route users into one of five specialized workflows, including QA, Billing, Refund, Cancel, and Review. For general QA, the system uses FAISS to search an in-memory vectorstore built exclusively from article metadata such as titles, summaries, and headers rather than raw article text. Retrieved articles are supplied to the language model as prompt context, followed by automated validations for safety, character limits, and hallucinated hyperlinks. In production A/B testing, this metadata-driven design doubled the chatbot resolution rate while achieving approximately 94% recall@5.
Shaping Product Understanding with Contrastive Reinforcement Learning
Etsy’s search and recommendation systems need product representations that capture distinguishing details—such as materials, craft, and visual style—beyond categories and titles, despite noisy listing data, varied inventory, and latency constraints. The approach fine-tunes the open-source Qwen3-VL-8B on query, engaged-listing, and non-engaged-listing triplets, generating candidate summaries and scoring them with a frozen two-tower dense retriever. Group Relative Policy Optimization (GRPO) uses the query–engaged versus query–non-engaged similarity margin as a joint reward, encouraging grounded summaries that separate alternatives rather than generic keyword stuffing. Human review found specific, accurate details beyond titles, while adding the summaries to Etsy’s Semantic Relevance model improved macro F1 by 8.7% relative to description n-grams. Online production experiments are planned.
Introducing the Ettin Reranker Family
Tom Aarsen introduces six Sentence Transformers CrossEncoder rerankers, ranging from 17M to 1B parameters, alongside their training data and full recipe. Each model uses an Ettin ModernBERT encoder with unpadded attention, RoPE, GeGLU, up to 8,192 tokens of context, and a modular classification head whose ablations favored CLS pooling over mean pooling. Training applies pointwise MSE distillation to scores from mixedbread-ai/mxbai-rerank-large-v2 using a broad and retrieval-focused dataset mixture. The conclusion reports state-of-the-art results at every released size, with the 400M model within 0.0024 and the 1B model within 0.0001 of the 1.54B teacher on MTEB, while all models outperform the ms-marco-MiniLM-L*-v2 family on MTEB and NanoBEIR.
Operator: A look under the hood
Operator is a customer-operations Agent built to help teams understand, manage, and improve customer experience across workspaces. The post distinguishes a production Agent from a prototype that simply gives an LLM API access and a system prompt, describing more than 50 purpose-built tools and 10 skills for data retrieval, reporting, investigation, and operational tasks. Its intelligence layer combines semantic search tuned against millions of support conversations, workspace-specific attribute awareness, and encoded reasoning, while the action layer supports configuration and content changes through reviewable diffs requiring explicit approval. A hybrid conversational and graphical interface presents charts, dashboards, and structured proposals within the existing platform. The post argues that read-only custom tools can be built in weeks, but production reliability, safe actions, multi-tenant isolation, graceful degradation, and ongoing maintenance require substantially greater investment.
Granite Embedding Multilingual R2: Open Apache 2.0 Multilingual Embeddings with 32K Context — Best Sub-100M Retrieval Quality
Granite Embedding Multilingual R2 introduces two Apache 2.0 multilingual embedding models built on ModernBERT: a 97M-parameter compact model and a 311M full-size model. Both cover more than 200 languages, receive enhanced retrieval training for 52 languages and nine programming languages, and support context lengths up to 32,768 tokens, compared with 512 for the R1 models. The compact model scores 60.3 on MTEB Multilingual Retrieval, while the full-size model scores 65.2; the reported results place them among the strongest open models in their parameter ranges. R2 replaces the XLM-RoBERTa architecture with alternating attention lengths, rotary position embeddings, newer multilingual and code tokenizers, and Flash Attention 2.0 support, while the 311M model adds Matryoshka embeddings and both models provide ONNX and OpenVINO weights.
Are you winning in AI search? Do this self-assessment.
AI search is reshaping consumer discovery, but the post says most brands do not know what AI platforms say about them. It presents a four-step assessment: choose one country, define a focused category, identify roughly 100 consumer questions, and test whether the brand appears in answers from ChatGPT, Gemini, Perplexity, and Google AI Overviews. Search Intelligence platforms track queries, mentions, recommendations, citations, and content gaps across search and AI tools. The post recommends answering real queries with dedicated content, earning coverage from accessible independent sources, participating helpfully in online conversations, and monitoring AI answers monthly because they change. Its examples report a 25% reduction in Orange call-center contacts, 50% lower Campari costs, and Aromazon reaching €40 million in annual revenue.
Full-screen Tabs and Compact Chats
This release introduces quality-of-life improvements to the Agents Window, centered on full-screen work areas and more compact agent conversations. Full screen expands files, changes, canvases, PRs, browsers, or terminals across the right panel, replaces chat with a floating prompt bar, and is available through the panel control, command palette, or Cmd/Ctrl+Shift+M. Compact chats offer Compact, Balanced, and Detailed tool-call density settings, ranging from concise results with minimal traces to near-complete step-by-step context. The release also improves PR review visibility, scrolling, streaming, context persistence, MCP authentication, model selection, search, and task status, while fixing reliability issues across PRs, background and cloud agents, MCP/OAuth, shortcuts, hooks, model routing, webhooks, and mirrored repositories.
ML based ranking using Nrtsearch
Nrtsearch’s Inference Plugin embeds machine-learning ranking in the Lucene-based search engine, addressing scalability problems in a two-stage design where a separate inference service scored candidates. Previously, large candidate sets and feature payloads created network-transfer and serialization overhead; the new flow extracts features from index documents or request parameters and scores documents within replica nodes. An ML Scorer selects models, transforms features, runs MLeap-based inference, and aggregates scores, with support for XGBoost, neural networks, multi-model execution, batch inference, and custom Java scorers. Models are loaded from MLflow through cluster configuration, while teams can use Lucene expressions or implement modular interfaces; testing, canary deployment, rolling updates, and Prometheus metrics support operations, and future work includes GPU-based ranking and additional inference engines.
Pushing the Frontier for Data Agents with Genie
Genie is Databricks’ data agent for complex questions across structured enterprise assets—tables, dashboards, and notebooks—and unstructured sources including workspace files, Google Drive, and Sharepoint. Unlike coding agents operating in static environments, it must discover relevant assets at enterprise scale, determine authoritative knowledge from potentially contradictory sources, and handle questions without verifiable tests or guaranteed answers. It combines specialized knowledge search using semantic context and metadata, parallel thinking across sampled trajectories, and Multi-LLM orchestration with optimized prompts for distinct sub-agents. On an internal benchmark of real-world data-analysis tasks, these techniques raised accuracy from 32% to over 90% against a leading coding agent while reducing cost and latency; specialized search alone improved table-discovery performance by up to 40%, while parallel thinking added latency and token costs before further optimization.
ClickHouse Release 26.4
ClickHouse Release 26.4 introduces 39 new features, 45 performance optimizations, and 238 bug fixes, focusing on broader SQL compatibility and indexing improvements. The update expands standard SQL support by allowing VALUES clauses as table expressions with column aliases, PostgreSQL-style EXTRACT units, NATURAL JOIN, and SET TIME ZONE commands. For text search, LIKE and ILIKE pattern queries using splitByNonAlpha tokenizers scan inverted index dictionaries instead of executing full-table scans when the dedicated evaluation setting is active. The release also provides the JSONAllValues function to extract all leaf values from a JSON column into a string array, enabling text skip indexes that significantly reduce scanned data and improve query latency.
From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest
Pinterest built a dedicated shopping candidate generation model to optimize for lower-funnel purchase conversions rather than relying strictly on engagement retrieval models. Because offsite conversion events are sparse and delayed, engineers trained a single multi-surface two-tower model using conversion data supplemented by duration-weighted click engagement and unengaged ad impressions as hard negatives. The two-tower retrieval architecture incorporates a parallel DCN v2 and MLP layer structure that allows simultaneous learning from the same inputs without information bottlenecks. Later revisions transitioned the system from a multi-head loss structure to a unified single-head multi-task architecture paired with an advertiser-level objective. Deployed across more than 600 million monthly active users, the system increased shopping conversion volume by 2.3% and boosted return on ad spend by 3.1% for United States shopping campaigns.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.