---
title: "Search"
description: "185 posts about Search, summarised, each linking to the original."
---

# Search
> 185 posts about Search, summarised, each linking to the original.

## Articles

### [How we optimized Dash's relevance judge with DSPy](https://yomu.fyi/post/how-we-optimized-dash-s-relevance-judge-with-dspy.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon
- Published: Mar 17, 2026

Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling.


### [GitTrends: A Google Trends style view of the GitHub ecosystem](https://yomu.fyi/post/gittrends-a-google-trends-style-view-of-the-github-ecosystem.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Lionel Palacin
- Published: Mar 10, 2026
- Type: Announcement

GitHub produces billions of events across issues, pull requests, and comments, but turning this vast stream into a real-time trends analyzer presents significant technical challenges. GitTrends is an open-source demo application designed to search and compare technology keywords across nearly ten billion GitHub events. Built to showcase the full-text search capabilities recently released in ClickHouse, the application indexes raw text without pre-computed answers or intermediate data transformation. ClickHouse executes full-text searches and aggregations within a single engine in one pass, eliminating cross-system joins and data movement. The demo includes live comparisons toggling between full-text search, bloom filters, and full table scans to illustrate query performance at scale.


### [Announcing General Availability of ClickHouse Full-text Search](https://yomu.fyi/post/announcing-general-availability-of-clickhouse-full-text-search.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Melvyn Peignon
- Published: Mar 10, 2026
- Type: Announcement

ClickHouse has reached general availability for its native Full-text Search feature, designed to accelerate token-based text filtering across large datasets. The capability relies on deterministic inverted indexes rather than probabilistic Bloom filter skip indexes, mapping individual tokens directly to row numbers. It supports plain strings, string arrays, and map keys or values while allowing configurable pre-processing and tokenization using standard SQL expressions. The system does not implement relevance scoring or store positional data for ranking, focusing instead on rapid filtering paired with large-scale analytical aggregations. Testing shows significant reductions in scanned granules alongside substantial latency improvements, though text indexes require more storage than Bloom filters.


### [Inside MongoDB Dublin: The Heart of Our International Growth](https://yomu.fyi/post/inside-mongodb-dublin-the-heart-of-our-international-growth.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: MongoDB
- Published: Feb 27, 2026
- Type: Announcement

Established in 2013 in Ballsbridge, MongoDB Dublin functions as the company's international headquarters and a primary regional engineering hub. The location employs over 300 individuals spanning more than 40 nationalities across Product and Technology, Sales, and Technical Services teams. Engineers at the hub take direct ownership of core database systems, tackling theoretical computer science problems in areas like query optimization and implementing capabilities like native vector search for embeddings. Benefits provided to employees include 27 days of annual leave, 20 weeks of paid parental leave, and access to internal employee resource groups such as MDBWomen, Queer Collective, and Config. The facility continues to expand its engineering teams to own critical product components rather than peripheral tasks.


### [Using LLMs to amplify human labeling and improve Dash search relevance](https://yomu.fyi/post/using-llms-to-amplify-human-labeling-and-improve-dash-search-relevance.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon,Dmitriy Meyerzon
- Published: Feb 26, 2026

Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time.


### [Innovating with MongoDB | Customer Successes, February 2026](https://yomu.fyi/post/innovating-with-mongodb-customer-successes-february-2026.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Katie Palmer
- Published: Feb 25, 2026
- Type: Problem & solution

Modern artificial intelligence startups frequently face operational bottlenecks when deploying autonomous agents and dynamic interfaces on rigid relational schemas or restricted database engines. To address these limitations, platforms including Modelence, Thesys, Emergent Labs, and Heidi integrated MongoDB Atlas as their operational data foundation. Emergent Labs eliminated recursive schema migration failures previously encountered with PostgreSQL by adopting document structures that match native JSON generation, supporting millions of deployed applications. Meanwhile, Heidi migrated from Amazon DocumentDB to utilize MongoDB Vector Search for retrieval-augmented generation pipelines under a single unified API without requiring external search databases. These architectural transitions streamlined continuous agent iterations, eliminated operational scaling downtime, and accelerated deployment cycles across diverse production environments.


### [Two years of vector search at Notion: 10x scale, 1/10th cost](https://yomu.fyi/post/two-years-of-vector-search-at-notion-10x-scale-1-10th-cost.md)
- Company: [Notion](https://yomu.fyi/company/notion.md)
- Author: Preeti Gondi, Mickey Liu, Nathan Louie, Calder Lund, Jacob Sager
- Published: Feb 19, 2026
- Type: Explainer

Notion describes two years of scaling vector search for AI Q&A, where semantic embeddings retrieve workspace and connected-tool content by meaning rather than exact keywords. The initial system combined Apache Spark batch indexing with Kafka-driven real-time updates, workspace-based sharding, and generation routing; rapid demand drove a 600x increase in daily onboarding capacity and cleared a waitlist of millions of workspaces by April 2024. A serverless migration cut vector database costs 50 percent from peak usage, while a later move to turbopuffer reduced search-engine spend 60 percent, lowered AWS EMR costs 35 percent, and improved p50 query latency from 70–100ms to 50–70ms. Further work uses xxHash-based page-state caching in DynamoDB to reprocess only changed spans and shifts embeddings generation from Spark and query embeddings away from a third-party API to Ray on Anyscale, with early rollout results pointing to a projected 90+ percent reduction in embeddings infrastructure costs.


### [Building a Movie Recommendation Engine with Hugging Face and Voyage AI](https://yomu.fyi/post/building-a-movie-recommendation-engine-with-hugging-face-and-voyage-ai.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Arek Borucki
- Published: Feb 17, 2026
- Type: Tutorial

Traditional movie search mechanisms depend on coarse filters such as genre, actor, or title, which fail to capture emotional context and narrative nuances. This guide demonstrates building a mood-based semantic recommendation system by combining the voyage-4-nano open-weights embedding model, Hugging Face datasets, and MongoDB Atlas Vector Search. The architecture uses Sentence Transformers to generate embeddings and exposes an endpoint through FastAPI to match user moods against movie plots. Truncating the Matryoshka embeddings from 2048 to 1024 dimensions balances semantic retrieval quality, storage footprint, and query latency. Testing shows that while abstract emotional queries return moderate similarity scores around 0.62 to 0.67, concrete descriptive queries exceed 0.75 without requiring exact keyword matches.


### [Securely indexing large codebases](https://yomu.fyi/post/securely-indexing-large-codebases.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Jeremy Stribling
- Published: Jan 27, 2026
- Type: Problem & solution

Cursor describes a secure way to index large codebases for semantic search, where naive processing can delay availability for hours. The system builds a Merkle tree of file and directory hashes, synchronizes only divergent branches, and caches embeddings by chunk content to reduce repeated work. For new users, a client-generated simhash finds a similar index within the same team or user's indexes, which is copied while background synchronization reconciles differences. Content proofs derived from the client's full Merkle tree filter search results so users see only files they can prove they possess. Reported time-to-first-query falls from 7.87 seconds to 525 milliseconds for the median repository, and from 4.03 hours to 21 seconds at the 99th percentile.


### [How Etsy Uses LLMs to Improve Search Relevance](https://yomu.fyi/post/how-etsy-uses-llms-to-improve-search-relevance.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Yuqing Zhang
- Published: Jan 16, 2026
- Type: Explainer

Etsy’s Search Relevance team addresses a limitation of engagement-based search signals: popular listings can attract clicks even when they are not the best match for a query. It introduces semantic relevance, measured through a framework combining human-curated golden labels, an LLM annotator, cascaded models, and production integrations, with categories for relevant, partially relevant, and irrelevant query-listing pairs. The data pipeline uses sampled search-log pairs, dual human annotation, an evolving guideline, few-shot chain-of-thought prompting with o3 in LangGraph, and self-consistency sampling; a Qwen 3 VL 4B teacher then labels data at scale before distilling into a BERT-based two-tower student. The student supports real-time use with less than 10ms additional latency, while the framework evaluates and enhances search relevance alongside engagement signals.


### [MongoDB.local San Francisco 2026: Ship Production AI, Faster](https://yomu.fyi/post/mongodb-local-san-francisco-2026-ship-production-ai-faster.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Peter Richards
- Published: Jan 15, 2026
- Type: Announcement

MongoDB announced several features aimed at streamlining AI application development across its database and search ecosystem. The Voyage 4 embedding model family is now generally available, introducing cross-model compatibility within a shared embedding space alongside the video-capable voyage-multimodal-3.5 model. For in-database operations, MongoDB Community Edition introduced Automated Embedding in public preview to eliminate external synchronization pipelines for vector generation. Vector Search gained Lexical Prefilters, replacing the knnBeta operator with a dedicated vectorSearch operator and vector data type inside $search aggregations. Additionally, MongoDB made its intelligent assistant generally available in Compass and released the source code for its mongot search engine under the SSPL.


### [Vision RAG: Enabling Search on Any Documents](https://yomu.fyi/post/vision-rag-enabling-search-on-any-documents.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Thibaut Gourdel
- Published: Jan 12, 2026
- Type: Tutorial

Traditional retrieval-augmented generation relies on text parsing and optical character recognition to extract data from complex visual documents. These legacy techniques introduce high engineering effort, inconsistent accuracy, and heavy scaling costs. Vision RAG resolves these challenges by embedding entire visual documents directly through multimodal models like Voyage AI's voyage-multimodal-3.5, removing the need for text extraction pipelines. In this pipeline, dense vector embeddings map both text queries and visual assets into a unified representation space. At query time, the system retrieves relevant document images and supplies them alongside the text prompt to Anthropic's Claude 3.5 Sonnet to generate visually grounded responses.


### [That’s a Wrap: MongoDB’s 2025 in Review & 2026 Predictions](https://yomu.fyi/post/that-s-a-wrap-mongodb-s-2025-in-review-2026-predictions.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Kevin O'Rourke
- Published: Dec 22, 2025
- Type: Announcement

MongoDB recapped its product releases, executive transitions, and enterprise adoption milestones throughout 2025 alongside predictions for 2026. The company acquired embedding and reranking specialist Voyage AI to mitigate large language model hallucinations and launched MongoDB AMP to accelerate legacy application modernization. Vector search and full-text search capabilities were expanded from MongoDB Atlas to Community Edition and Enterprise Server, enabling on-premises and hybrid retrieval-augmented generation workloads. Enterprise implementations included pharmaceutical distributor McKesson scaling tracking infrastructure to handle 1.2 billion containers annually after replacing its monolithic legacy system. Executive predictions for 2026 emphasized human-in-the-loop verification, grounding language models with factual retrieval, and building multi-cloud resilience against outages and data sovereignty constraints.


### [Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries](https://yomu.fyi/post/token-count-based-batching-faster-cheaper-embedding-inference-for-quer.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Chengcheng Pei, Yuan Lin
- Published: Dec 18, 2025
- Type: Problem & solution

Serving embedding models for short search queries often suffers from poor GPU efficiency because traffic is spiky and memory-bound. Traditional time-window and request-count batching strategies lead to inconsistent GPU utilization, while tensor padding wastes compute on empty tokens. To resolve these bottlenecks, Voyage AI implemented token-count-based batching paired with padding removal in inference engines like vLLM. The architecture uses Redis with Lua scripts to atomically aggregate pending requests until reaching an optimal hardware saturation token threshold. Across production deployments, this approach achieved a 50% reduction in GPU inference latency with 3X fewer GPUs and improved throughput by up to 8×.


### [Tangle: An open-source ML experimentation platform built for scale (2025) - Shopify](https://yomu.fyi/post/tangle-an-open-source-ml-experimentation-platform-built-for-scale-2025.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 5, 2025
- Type: Announcement

Shopify introduces Tangle, an open-source, platform-agnostic machine-learning experimentation platform built for Search and Discovery workflows at commerce scale. It targets six stated failure modes, including repeated data preparation, poor reproducibility, slow deployment, unstructured notebooks, manual query tracking, and limited sharing. Users compose visual directed acyclic graph pipelines from reusable YAML components that wrap containerized or inline command-line programs in multiple languages, while file-based execution moves artifacts through storage. Tangle’s content-based caching reuses results when output content hashes remain identical, including artifacts from still-running executions, rather than relying only on lineage changes. Shopify reports a ten-hour pipeline completing in twenty minutes after one component changes, more than a year of compute time savings, and production use across ranking, semantic search, recommendation, and feature engineering workloads.


### [Building Deep Research: How we Achieved State of the Art](https://yomu.fyi/post/building-deep-research-how-we-achieved-state-of-the-art.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Michael Griff, Dean Sacoransky, Noah Nefsky
- Published: Nov 24, 2025
- Type: Problem & solution

Building production AI research agents presents challenges around context window pollution, escalating token costs, and architectural brittleness across model updates. To resolve these issues, Tavily rebuilt its deep research system around simplified orchestration, compact tooling, and active context curation. The agent mimics human research workflows by distilling tool outputs into concise reflections for ongoing reasoning, withholding raw retrieved web data until the final deliverable stage. Compared to traditional ReAct propagation architectures where token consumption scales quadratically, this reflection-based approach achieves linear token growth. As a result, the system reduced token consumption by 66% compared to Open Deep Research while reaching state-of-the-art performance on DeepResearch Bench.


### [Building world-class product search at Shopify: Where C++ excellence meets ML innovation](https://yomu.fyi/post/building-world-class-product-search-at-shopify-where-c-excellence-meet.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 12, 2025
- Type: Problem & solution

Shopify describes a product-search stack built for multi-tenant commerce, where real-time indexing, retrieval, ranking, experimentation, and merchant controls must operate at global scale. Commerce ranking combines classical information retrieval with transformers, neural rankers, LightGBM, Catboost, and signals such as relevance, purchase popularity, brand trust, and navigational intent, while supporting privacy, multi-language, and accessibility requirements. RankFlow, a Python-like domain-specific language, lets data scientists change ranking logic without C++ expertise, while TurboDSL compiles and executes it with type safety, vectorized operations, cache optimization, predictable latency, and efficient memory use. The team first shipped SimScorerDSL in C++, then built TurboDSL without changing the DSL syntax, enabling continued experimentation during the transition. TurboDSL achieved a 48% speedup in ranking feature computation, and automated pull-request analysis compares component timings, statistical significance within a ±2% tolerance, and side-by-side results to detect performance or accuracy regressions.


### [Improving agent with semantic search](https://yomu.fyi/post/improving-agent-with-semantic-search.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Stefan Heule, Emily Jia & Naman Jain
- Published: Nov 6, 2025
- Type: Benchmark

Cursor describes semantic search as a complement to grep for helping coding agents locate relevant code in large codebases. Its evaluation used Cursor Context Bench across widely used models, comparing tool sets with and without semantic search, and found an average 12.5% accuracy improvement, with gains ranging from 6.5% to 23.5%. In an online A/B test, semantic search increased code retention by 0.3% overall and 2.6% on codebases with at least 1,000 files, while its absence produced a 2.2% increase in dissatisfied follow-up requests. The retrieval system uses a custom embedding model trained from agent session traces, with an LLM ranking helpful content and training aligning embedding similarity scores to those rankings. The conclusion is that combining semantic search with grep produces the best outcomes, especially for large codebases.


### [Sentence Transformers is joining Hugging Face!](https://yomu.fyi/post/sentence-transformers-is-joining-hugging-face.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tom Aarsen
- Published: Oct 22, 2025
- Type: Announcement

Sentence Transformers is officially transitioning its stewardship from the Ubiquitous Knowledge Processing Lab at TU Darmstadt to Hugging Face. The open-source embedding library was introduced in 2019 by Nils Reimers using a Siamese network architecture to overcome standard BERT limitations on sentence-level semantics. Tom Aarsen of Hugging Face has maintained the project since late 2023 and will continue leading its development. Under Hugging Face, the project retains its Apache 2.0 license and leverages infrastructure such as continuous integration and automated testing. Sentence Transformers currently serves over one million monthly unique users, with more than 16,000 community models publicly hosted on the Hugging Face Hub.


### [Supercharge your OCR Pipelines with Open Models](https://yomu.fyi/post/supercharge-your-ocr-pipelines-with-open-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: merve, Aritra Roy Gosthipaty, Daniel van Strien, Hynek Kydlicek, Andres Marafioti, Vaibhav Srivastav, Pedro Cuenca
- Published: Oct 21, 2025
- Type: Explainer

Modern vision-language OCR models can transcribe handwritten text, multiple scripts, mathematical expressions, and chemical formulas while also interpreting tables, charts, and images. The guide compares open OCR models and explains how output choices—including DocTags, HTML, Markdown, and structured JSON—affect digital reconstruction, LLM input, and programmatic processing. It also covers locality awareness through anchors or bounding boxes, prompt-based task switching, fine-tuning versus out-of-the-box use, and factors for selecting a model. Practical deployment options include local or managed inference, vLLM, Hugging Face Endpoints, and batch scripts that add OCR results to datasets. The discussion extends document AI beyond transcription to visual retrieval and direct document question answering with vision-language models, especially where converted text can lose layout or visual context.


[Newer posts](https://yomu.fyi/topic/search/page/5.md) · [Older posts](https://yomu.fyi/topic/search/page/7.md)