---
title: "Search"
description: "185 posts about Search, summarised, each linking to the original."
---

# Search
> 185 posts about Search, summarised, each linking to the original.

## Articles

### [ML based ranking using Nrtsearch](https://yomu.fyi/post/ml-based-ranking-using-nrtsearch.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Mohammad Mohtasham (Software Engineer); Tao Yu (Software Engineer)
- Published: May 11, 2026
- Type: Explainer

Nrtsearch’s Inference Plugin embeds machine-learning ranking in the Lucene-based search engine, addressing scalability problems in a two-stage design where a separate inference service scored candidates. Previously, large candidate sets and feature payloads created network-transfer and serialization overhead; the new flow extracts features from index documents or request parameters and scores documents within replica nodes. An ML Scorer selects models, transforms features, runs MLeap-based inference, and aggregates scores, with support for XGBoost, neural networks, multi-model execution, batch inference, and custom Java scorers. Models are loaded from MLflow through cluster configuration, while teams can use Lucene expressions or implement modular interfaces; testing, canary deployment, rolling updates, and Prometheus metrics support operations, and future work includes GPU-based ranking and additional inference engines.


### [Pushing the Frontier for Data Agents with Genie](https://yomu.fyi/post/pushing-the-frontier-for-data-agents-with-genie.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: May 8, 2026
- Type: Benchmark

Genie is Databricks’ data agent for complex questions across structured enterprise assets—tables, dashboards, and notebooks—and unstructured sources including workspace files, Google Drive, and Sharepoint. Unlike coding agents operating in static environments, it must discover relevant assets at enterprise scale, determine authoritative knowledge from potentially contradictory sources, and handle questions without verifiable tests or guaranteed answers. It combines specialized knowledge search using semantic context and metadata, parallel thinking across sampled trajectories, and Multi-LLM orchestration with optimized prompts for distinct sub-agents. On an internal benchmark of real-world data-analysis tasks, these techniques raised accuracy from 32% to over 90% against a leading coding agent while reducing cost and latency; specialized search alone improved table-discovery performance by up to 40%, while parallel thinking added latency and token costs before further optimization.


### [ClickHouse Release 26.4](https://yomu.fyi/post/clickhouse-release-26-4.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 8, 2026
- Type: Announcement

ClickHouse Release 26.4 introduces 39 new features, 45 performance optimizations, and 238 bug fixes, focusing on broader SQL compatibility and indexing improvements. The update expands standard SQL support by allowing VALUES clauses as table expressions with column aliases, PostgreSQL-style EXTRACT units, NATURAL JOIN, and SET TIME ZONE commands. For text search, LIKE and ILIKE pattern queries using splitByNonAlpha tokenizers scan inverted index dictionaries instead of executing full-table scans when the dedicated evaluation setting is active. The release also provides the JSONAllValues function to extract all leaf values from a JSON column into a string array, enabling text skip indexes that significantly reduce scanned data and improve query latency.


### [From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest](https://yomu.fyi/post/from-clicks-to-conversions-architecting-shopping-conversion-candidate.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: Apr 27, 2026
- Type: Problem & solution

Pinterest built a dedicated shopping candidate generation model to optimize for lower-funnel purchase conversions rather than relying strictly on engagement retrieval models. Because offsite conversion events are sparse and delayed, engineers trained a single multi-surface two-tower model using conversion data supplemented by duration-weighted click engagement and unengaged ad impressions as hard negatives. The two-tower retrieval architecture incorporates a parallel DCN v2 and MLP layer structure that allows simultaneous learning from the same inputs without information bottlenecks. Later revisions transitioned the system from a multi-head loss structure to a unified single-head multi-task architecture paired with an advertiser-level objective. Deployed across more than 600 million monthly active users, the system increased shopping conversion volume by 2.3% and boosted return on ad spend by 3.1% for United States shopping campaigns.


### [Do you still need Elasticsearch for log analytics? ClickHouse says no.](https://yomu.fyi/post/do-you-still-need-elasticsearch-for-log-analytics-clickhouse-says-no.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tom Schreiber, Lionel Palacin
- Published: Apr 23, 2026
- Type: Benchmark

ClickHouse introduced a redesigned full-text search powered by inverted indexes that feeds matching documents directly into its vectorized analytical engine for observability and log analytics workloads. To evaluate this capability, the open-source versions of ClickHouse and Elasticsearch were benchmarked on single-node AWS EC2 instances across synthetic OpenTelemetry log datasets scaling up to 50 billion rows. The benchmark evaluated nine representative query patterns combining text search with filtering, grouping, counting, and time-series aggregations under both cold and hot cache conditions. ClickHouse achieved a roughly fivefold smaller storage footprint on disk while running cold queries four to six times faster and hot queries approximately 1.7 to 2.6 times faster than Elasticsearch. Furthermore, ClickHouse ingested the 50-billion-row dataset in under four hours out of the box, whereas the tuned Elasticsearch setup required approximately five days.


### [How to Influence LLM Responses: A Guide to AI Visibility](https://yomu.fyi/post/how-to-influence-llm-responses-a-guide-to-ai-visibility.md)
- Company: [Brandwatch](https://yomu.fyi/company/brandwatch.md)
- Author: Ksenia Newton
- Published: Apr 22, 2026
- Type: Explainer

AI tools increasingly answer consumer questions directly, reducing click-throughs while making remaining referrals more likely to convert, according to the Post. Brands therefore need to influence the broader information environment that shapes LLM recommendations, not merely optimize the URLs shown as citations. LLMs draw on owned content, earned media, and social conversations, including crawlable trade publications, industry blogs, Reddit, forums, and podcast transcripts. A telecommunications brand illustrates the owned-content route: after identifying real troubleshooting questions and publishing targeted answers, it reduced call center contact volume by 25% and improved issue resolution. The guidance also stresses concrete claims, authentic participation, and monitoring outdated or inaccurate narratives to build fresher context around a brand.


### [What is pgvector?](https://yomu.fyi/post/what-is-pgvector.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 17, 2026
- Type: Explainer

pgvector is an open-source PostgreSQL extension that adds vector storage, indexing, and similarity search for embeddings alongside relational application data. It supports semantic search, recommendations, retrieval-augmented generation (RAG), image similarity, anomaly detection, and deduplication without requiring a separate vector database for many workloads. The extension uses the vector data type and distance metrics including L2, cosine similarity, and inner product, with HNSW and IVFFlat indexes offering different speed and memory trade-offs. Vector queries can also combine relational filters with Postgres full-text search for hybrid retrieval. pgvector is positioned as a practical choice for existing Postgres deployments, while pgvectorscale can extend its usefulness as data and query demands grow; dedicated vector databases become more relevant at larger scales, and Databricks AI Search serves complementary lakehouse workloads.


### [April 2026 newsletter](https://yomu.fyi/post/april-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: Apr 16, 2026
- Type: Announcement

ClickHouse 26.3 LTS introduces experimental materialized common table expressions, WebAssembly user-defined functions, and enables asynchronous inserts by default. Alongside the release, Alasdair Brown launched clickhousectl to manage local installations and cloud deployments with built-in agentic development features and authentication safeguards. User case studies show Goldsky accelerating historical blockchain backfills by twelve times through direct ClickHouse reads using Apache Arrow, replacing their prior Kafka and Avro architecture. Additionally, the engineering team redesigned the full-text search index with front-coding compression, a sparse lookup index, and adaptive posting lists to handle expensive random reads on object storage. ClickHouse also previewed community architecture patterns for lightweight SIEM setups using Clickdetect and Wazuh log collection.


### [Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents](https://yomu.fyi/post/ecom-rlve-adaptive-verifiable-environments-for-e-commerce-conversation.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Rahul Bajaj, Jaya Nupur, Anuj Garg, ben burtenshaw
- Published: Apr 16, 2026
- Type: Explainer

Ecom-RLVE extends the RLVE framework from single-turn reasoning puzzles to multi-turn, tool-augmented e-commerce conversations. EcomRLVE-GYM supplies eight environments spanning product discovery, substitution, cart building, returns, order tracking, policy QA, bundle planning, and multi-intent journeys, with procedural problem generation, a 12-axis difficulty curriculum, and algorithmically verifiable rewards. Each episode requires tool use and world-state changes; rewards combine task completion, efficiency, and a penalty for recommending product IDs not retrieved, while malformed outputs and illegal calls fail immediately. Adaptive scheduling advances each environment independently when its success rate is reliable. An initial study trained Qwen 3 8B with DAPO for 300 steps on cart building and reported progressively increasing difficulty reached, consistent with a steady learning signal.


### [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://yomu.fyi/post/training-and-finetuning-multimodal-embedding-reranker-models-with-sent.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tom Aarsen
- Published: Apr 16, 2026
- Type: Tutorial

Sentence Transformers provides a Python library for training and using embedding and reranker models across text, images, audio, and video in retrieval and semantic search applications. The post shows how to finetune multimodal models on task-specific data, using Visual Document Retrieval as the example: matching text queries with document page images while preserving charts, tables, and layout. It describes the SentenceTransformerTrainer pipeline, model and processor configuration, dataset handling, and alternative Router-based encoders, while also covering multimodal reranker training with BinaryCrossEntropyLoss and direction-specific datasets. On its evaluation data, finetuning Qwen/Qwen3-VL-Embedding-2B raised NDCG@10 from 0.888 to 0.947 and outperformed all tested VDR models, including models up to 4x its size.


### [Agentic reasoning in practice: Making sense of structured and unstructured data](https://yomu.fyi/post/agentic-reasoning-in-practice-making-sense-of-structured-and-unstructu.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: Apr 14, 2026
- Type: Benchmark

Enterprise questions that connect structured records with unstructured text are difficult for single-turn retrieval systems to answer reliably. The post evaluates Databricks Agent Bricks Supervisor Agent (SA), a declarative agent builder built on the internal aroll framework, across STaRK’s Amazon, MAG, and Prime domains and six KARLBench tasks. SA decomposes questions, routes sub-questions to Genie or Knowledge Assistant, combines results, and can reformulate queries or use SQL JOINs when initial retrieval is incomplete. It outperforms rerun SoTA baselines by 4% on STaRK-Amazon, 21% on MAG, and 38% on Prime, while delivering a 23% improvement on FinanceBench and consistent gains across KARLBench; the authors attribute the results to multi-step tool selection, reflection, and precise instructions rather than custom code.


### [Building real-time product search on Databricks](https://yomu.fyi/post/building-real-time-product-search-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jiayi Wu, Luke Lefebure, Adam Gurary
- Published: Apr 14, 2026
- Type: Explainer

Real-time product search for an online marketplace must retrieve, filter, rank, and respond in milliseconds while balancing latency, relevance, revenue, and click-through rate. The Databricks architecture divides the pipeline into ingestion, retrieval, and refinement, using Lakeflow pipelines and AI Functions for source processing, AI Search for embeddings and hybrid retrieval, and Lakebase for sub-10ms operational signals. It recommends validating p95/p99 latency, throughput, retrieval quality, and engagement metrics, with MLflow supporting systematic evaluation and caching helping meet strict latency requirements. As an example, FOX Sports used AI Search for thousands of QPS and achieved a 2x improvement in query success rate, combining streaming ingestion, two-phase retrieval, model serving, caching, and trending searches.


### [Multimodal Embedding & Reranker Models with Sentence Transformers](https://yomu.fyi/post/multimodal-embedding-reranker-models-with-sentence-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tom Aarsen
- Published: Apr 9, 2026
- Type: Tutorial

Sentence Transformers v5.4 adds multimodal embedding and reranking support for text, images, audio, and video through its Python API. Embedding models map modalities into a shared space, while multimodal Cross Encoder rerankers score relevance for mixed-modality pairs, enabling cross-modal similarity, retrieval, and multimodal RAG workflows. The guide shows model loading, URL-based image encoding, encode\_query() and encode\_document() prompt handling, similarity calculation, and ranking, with a Qwen3-VL-Embedding-2B example producing 2,048-dimensional image vectors and matching text queries to images. It notes that cross-modal scores can be lower because of the modality gap, while relative ordering can still support retrieval, and that rerankers generally offer higher quality at the cost of slower per-pair processing; installation extras and GPU memory requirements vary by modality and model.


### [ClickHouse at FOSDEM 2026](https://yomu.fyi/post/clickhouse-at-fosdem-2026.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tyler Hannan
- Published: Apr 8, 2026
- Type: Announcement

The ClickHouse community and development team participated in FOSDEM 2026 in Brussels, hosting a community dinner and delivering technical presentations across multiple developer tracks. Presentations covered a range of engineering topics, including building geospatial analytical applications with Leaflet and ClickHouse, as well as integrating LLVM XRay to dynamically hot-patch production logging without redeployments. Additional ClickHouse sessions addressed the engineering challenges of integrating Rust libraries into large C++ codebases, optimizing low-level memcpy routines, and implementing specialized inverted indexes for large-scale text processing. Contributed community sessions detailed workflows for upstream patches in MariaDB and Postgres, alongside WebShield, an open-source Python daemon that streams web accelerator access logs into ClickHouse for bot analysis.


### [Building Biz Ask Anything: From Prototype to Product](https://yomu.fyi/post/building-biz-ask-anything-from-prototype-to-product.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Maria Christoforaki, Group Tech Lead; Shree Shalini Pusapati, Software Engineer
- Published: Mar 27, 2026
- Type: Problem & solution

Yelp expanded Yelp Assistant to business pages by developing a question-answering architecture that extracts concise, evidence-backed answers from reviews, photos, and structured facts. The production system replaces static snapshots with near-real-time streaming pipelines for reviews and structured properties alongside weekly batch ingestion for menus and website data. For each incoming query, the system evaluates intent, retrieves relevant context through keyword and embedding searches, and constructs targeted prompts with dynamically selected few-shot examples. Synthesized answers are augmented with supporting visuals and citations before being streamed token-by-token to users. Generating suggested questions directly from specific business content rather than generic categories improved user engagement by roughly fifty percent and reduced unanswerable queries by twenty-six percent.


### [Building high-performance full-text search for object storage](https://yomu.fyi/post/building-high-performance-full-text-search-for-object-storage.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Elmi Ahmadov, Jimmy Aguilar, and George Larionov
- Published: Mar 24, 2026
- Type: Problem & solution

ClickHouse redesigned its native full-text index to address latency bottlenecks caused by random read patterns on remote object storage. Instead of using a Finite State Transducer dictionary, the new architecture organizes indexed tokens into fixed-size sorted blocks compressed with front-coding. Each data part separates index data across a small in-memory sparse index file, a sequential dictionary file, and a posting list file. This layout allows the query engine to pinpoint token locations with minimal random I/O and resolve multiple full-text functions directly from index files. Consequently, ClickHouse Cloud maintains predictable search latency across shared object storage while matching local disk performance and scaling through parallel query execution.


### [Fast regex search: indexing text for agent tools](https://yomu.fyi/post/fast-regex-search-indexing-text-for-agent-tools.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Vicent Marti
- Published: Mar 23, 2026
- Type: Explainer

Agentic coding has revived grep as a critical way to gather source-code context, but ripgrep can stall on large monorepos because it scans every file. The proposed remedy is a local inverted index of overlapping character trigrams, with regular expressions parsed into required and alternative trigram constraints that narrow candidate files before exact matching. The index stores posting lists in one disk file and a sorted hash-to-offset lookup table in another, memory-mapping only the table for binary-search lookups; hash collisions can broaden results but cannot produce incorrect ones. Freshness is prioritized because agents often search for text they have just written, and the client index layers user and agent changes over a Git commit. The reported conclusion is that instant local search creates qualitative workflow improvements, especially for bug investigation and large Enterprise repositories.


### [Build a Domain-Specific Embedding Model in Under a Day](https://yomu.fyi/post/build-a-domain-specific-embedding-model-in-under-a-day.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Steve Han, Rucha Apte, Sean Sodha, Oliver Holworthy
- Published: Mar 20, 2026
- Type: Tutorial

The tutorial presents a pipeline for adapting a general-purpose embedding model when it misses fine-grained distinctions in domain documents and labeled query–document data is unavailable. It generates synthetic question–answer pairs with NeMo Data Designer, filters them for quality, mines hard negatives with a 95% positive-score margin ceiling, and unrolls multi-hop queries for contrastive training. The recipe fine-tunes Llama-Nemotron-Embed-1B-v2, evaluates base and adapted models with BEIR metrics, and exports the result to ONNX or TensorRT. On the reported datasets, it improved Recall@10 and NDCG@10 by more than 10%; Atlassian increased Recall@60 from 0.751 to 0.951 on JIRA data. The complete workflow can run in roughly 2–3 hours for about 500 documents, using a single 80GB Ampere-or-newer GPU.


### [March 2026 newsletter](https://yomu.fyi/post/march-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: Mar 19, 2026
- Type: Announcement

The March 2026 ClickHouse newsletter details major platform updates, community deployments, and architectural practices for high-throughput analytics. Recent product announcements include ClickHouse 26.2 featuring time-based block flushing for streaming feeds, general availability of native inverted indexes for full-text search, and the release of chDB 4 with a Pandas-compatible DataStore API. Community implementations show Cloudflare managing 1.61 quadrillion daily events, while Trigger.dev isolated multi-tenant SQL execution using a domain-specific query language that compiles safe ClickHouse queries. In data optimization techniques, Hookdeck accelerated webhook payload search from over 30 seconds to under 400 milliseconds by partitioning payload hashes into typed bucket columns and scanning iterative time windows. Additional operational guidance emphasizes setting the ttl\_only\_drop\_parts parameter and aligning table partitioning with time units so background merges drop entire data parts cleanly without triggering mutation rewrites.


### [How ClickStack makes ClickHouse faster for observability](https://yomu.fyi/post/how-clickstack-makes-clickhouse-faster-for-observability.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mike Shi
- Published: Mar 18, 2026
- Type: Problem & solution

ClickHouse delivers high performance for telemetry data, but unoptimized queries on high-cardinality observability workloads can bypass pruning, inflate intermediate state, and exhaust cluster resources. ClickStack addresses this challenge by embedding engine-specific optimization best practices directly into how queries are generated, structured, and executed. For search workflows, the platform executes progressive queries across bounded time windows using optimize\_read\_in\_order to return initial rows rapidly without scanning entire historical ranges. ClickStack also segments large chart aggregations into parallel bucket-aligned queries, automatically redirects map attribute filters to pre-materialized columns, and extends lazy materialization thresholds to minimize runtime input-output overhead. These query formulation strategies are being packaged into specialized analytical endpoints to power notebooks and external tools without requiring manual SQL tuning or exposing raw database interfaces.


[Newer posts](https://yomu.fyi/topic/search/page/4.md) · [Older posts](https://yomu.fyi/topic/search/page/6.md)