Bloom Filters: The Cheap Gate Before the Expensive Question
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
Loading…
Topic
Search
185 posts about Search. Every summary links to the original.
- 185 posts
- 31 companies
- 3 explainers
See the idea
Search explainers
Apache Lucene Scoring: BM25 Relevance vs StableTfl Consistency
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
AI Search: Speed vs. Relevance Trade-off
AI Search balances retrieval speed and result quality by choosing between dense embeddings, sparse lexical search, and hybrid approaches. Understand which trade-off fits your workload.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Storing OpenAI embeddings in Postgres with pgvector
Post describes how embeddings represent semantic relatedness as vectors and how OpenAI’s text-embedding-ada-002 model produces 1536-dimensional embeddings for text. It addresses storing and searching large or dynamically changing collections that may not fit comfortably in memory, using PostgreSQL and the pgvector extension instead of a small CSV-and-NumPy workflow. The implementation enables the vector extension, creates a documents table with a vector(1536) column, and defines match_documents() with cosine distance, thresholds, counts, and ordering. It also shows JavaScript code for generating embeddings, a search flow that retrieves matching documents for prompt context, and token limiting before a completion request, while noting that documents can be chunked. For larger tables, the post recommends an ivfflat index with vector_cosine_ops and gives 4 * sqrt(table_rows) as a starting point for lists.
Launch Week 6: Wrap Up
Supabase’s Launch Week Day 6 wrap-up consolidates the products and platform changes announced during the event. The roundup announces redesigned Next.js documentation, Storage v2 features for image resizing, webhooks, and a Smart CDN, plus multi-factor authentication usable in Row Level Security policies. It also introduces Wrappers, an Alpha framework for connecting Postgres to external systems, Vault for encrypted secrets, and pg_graphql 1.0 with general availability. Additional releases include dashboard-enabled point-in-time recovery for Pro Projects, custom domains, pg_crdt, PostgreSQL 15 for new projects, a PostgREST 11 pre-release, and the PGroonga multilingual full-text search extension. Community partnerships with Flutterflow and OneSignal provide Flutter app development and push-notification integrations.
Query expansion based on user behaviour
Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.
Postgres Full Text Search vs the rest
The post compares PostgreSQL Full Text Search with dedicated search engines including MeiliSearch, OpenSearch, SQLite FTS, and Typesense, asking whether a separate system is justified for common application search. It presents a PostgreSQL setup using a generated stored column built with to_tsvector('english', ...) and a GIN index, queried with websearch_to_tsquery. The benchmark loads a movie dataset and focuses on query speed after data ingestion, using comparable scripts and default configurations with limited tuning. PostgreSQL was close in performance to several alternatives; in-memory SQLite was fastest, while Typesense and MeiliSearch handled misspellings correctly and OpenSearch ingested data quickly. The conclusion is that PostgreSQL can provide good-enough search quality and speed for basic cases without adding another system, although dedicated engines offer capabilities PostgreSQL does not.
Typeahead Search at Nextdoor
Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.
Liftoff! How to get started with your first ML project 🚀
Beginners in machine learning frequently face difficulties when selecting a framework and defining the scope of their initial hands-on project. Sentence Transformers provides an accessible starting point by computing dense vector representations for sentences, paragraphs, and images, enabling semantic search through cosine similarity. A structured four-step strategy guides practitioners through listing library capabilities, identifying compelling datasets, selecting a familiar secondary tool, and brainstorming project concepts. Applying this framework, a song lyrics dataset was paired with Gradio Blocks to build a prompt-based playlist generator. Developing this application required evaluating pre-trained models, hosting generated embeddings on Hugging Face Spaces, and utilizing multi-processor support to accelerate embedding generation.
Getting Started With Embeddings
Embeddings represent unstructured information such as text and images as numerical vectors in a shared semantic space. To demonstrate their utility, a simple semantic search engine is built over US Social Security Medicare frequently asked questions. The system generates 384-dimensional vector representations for thirteen FAQ entries by dispatching POST requests to the Hugging Face Inference API using the sentence-transformers/all-MiniLM-L6-v2 model. Incoming user queries are converted into matching vector representations and evaluated against stored dataset vectors using the util.semantic_search function from the Sentence Transformers library. By calculating cosine similarity scores, the system retrieves and ranks the five most semantically relevant questions without requiring custom keyword rules or massive labeled training sets.
How facial recognition technology keeps you safe
Grab utilizes facial recognition technology across its platform for driver authentication, passenger verification, and digital electronic Know Your Customer (e-KYC) processes. The core pipeline consists of image preprocessing through face detection and alignment, anti-spoofing checks, feature extraction into high-dimensional vector embeddings, and downstream verification or search. To counter spoof attacks like screen replays, synthetic moiré patterns are generated and cropped face patches are used during training and inference to focus on local structures rather than global semantic noise. Face verification challenges involving shallow ID datasets and masked faces are resolved using semi-Siamese training architectures and masked data augmentations.
Search architecture revamp
Before 2021, Grab’s Deliveries search architecture used an inverted index for textual matching, leaving it unable to account for user intention, context, or non-textual recall methods. Queries such as “Roti Prata” could prioritize merchant names over dishes, while the same query could imply different objectives by time or location; merging Ads and organic results at the Deliveries API layer also discarded recall-stage data. The revamped flow adds boosting, multi-recall, and ranking, enabling intent recognition, query expansion, embedding-based and graph-based recall, and joint ranking across business verticals. A platform-defined framework assembles service-team plugins through configuration, separating core flow from application logic and allowing components such as Ads and Mart recall to be reused across Food, Mart, and universal search flows. The resulting platform provides intent recognition, semantic search, mixed Ads-and-organic ranking, and simpler onboarding, while making new search modules and use cases easier to add.
Image search with 🤗 datasets
Hugging Face datasets expanded its capabilities by introducing an Image feature type, enabling image processing and integration with vector indexing tools. The library was applied to a sample of historical book embellishments extracted via OCR from the British Library. Images were loaded using the ImageFolder loader, enriched with filename metadata, and pushed to the Hugging Face Hub. A FAISS index and CLIP embeddings were used to retrieve images matching natural language queries, such as categories, specific objects, and boolean operators. While the retrieval system demonstrated reasonable semantic search results across various prompts, full public deployment was avoided due to CLIP model card restrictions and potential bias in the historical dataset.
BERT 101 - State Of The Art NLP Model Explained
Developed in 2018 by Google AI Language, Bidirectional Encoder Representations from Transformers addresses the historical challenge of machines lacking contextual understanding of human language. The model relies on an encoder-only Transformer architecture pre-trained on a 3.3-billion-word corpus consisting of Wikipedia and Google BooksCorpus. Training simultaneously combines masked language modeling, which hides 15% of tokenized words to enforce bidirectional context learning, with next sentence prediction across balanced sentence pairs. Pre-trained on Cloud TPUs over four days, BERT unifies solutions for more than eleven common NLP tasks and can be fine-tuned on task-specific annotated data within minutes. Unmasking experiments demonstrate that the model can also inherit distinct societal and gender biases from its underlying training corpora when predicting professions.
Supercharged Searching on the 🤗 Hub
Programmatically searching the Hugging Face Hub previously required navigating web browser widgets or enduring trial and error to guess exact query string formats. The huggingface_hub library solves this issue by introducing helper utilities such as ModelSearchArguments, DatasetSearchArguments, and ModelFilter alongside HfApi. These namespace helpers translate accessible Python attributes into the formatted parameters expected by the backend API, covering datasets, tasks, and libraries. For complex queries across multiple tasks, frameworks, and datasets, developers can combine criteria inside ModelFilter instances and retrieve matching model metadata via api.list_models. Under the hood, the library uses AttributeDictionary, a data structure inspired by fastcore that enables nested tab-completion while supporting standard dictionary key indexing for special characters.
Search at Shopify—Range in Data and Engineering is the Future
Search at Shopify is presented as a case for range across data science and engineering, rather than narrow specialization in either discipline. The article describes a team that treats work as “search” instead of separating data and engineering responsibilities: engineers learn experimentation and model evaluation, while data scientists build performant, testable, maintainable production systems. It frames search as a sequence of trade-offs among relevance, correctness, scalability, performance, stability, and maintainability, arguing that people who understand only half the constraints can produce siloed handoffs, deployment delays, or suboptimal systems. Shopify’s stated model is a single collaborative team with shared planning and execution, intended to reduce territorial boundaries and support smart, fast, scalable search. The article extends this principle to personalized and conversational products, concluding that future work will require practitioners to move between data and engineering.
Nerd Out on 10 of Our Favorite Posts From 2021
Shopify’s year-end roundup curates ten favorite engineering posts from 2021, spanning Ruby, storefronts, data analysis, databases, search, GraphQL, CI, APIs, and mobile development, while also noting several older reader favorites. The selection includes tutorials on building a web app with standard Ruby libraries, exploratory data analysis, GraphQL, Apache Beam for search, and React Native App Clips, alongside technical accounts of Hydrogen, YJIT, MySQL upgrades, and GraphQL rate limiting. YJIT was merged upstream and included in Ruby 3.1.0, while the CI team reduced the p95 of Shopify’s core monolith from 45 minutes to 18 minutes. The roundup also describes query-cost-based GraphQL rate limiting, rollback-testing roadblocks during a MySQL upgrade, and an App Clip effort that used a fail-fast, iterative approach.
Train a Sentence Embedding Model with 1B Training Pairs
Sentence embeddings map sentences into dense vectors to capture semantic meaning for downstream tasks such as clustering, asymmetric question answering, and text search. To train general-purpose models at scale, the team utilized a dataset compiled from multiple sources containing up to one billion sentence pairs. Training leveraged contrastive learning with Multiple Negative Ranking Loss and in-batch negatives, running across seven TPU v3-8 accelerators with JAX and Flax. Batches combined samples from at least two datasets to balance hard negatives within topics against broader cross-topic semantic structure. The project resulted in twenty trained models—including variants of RoBERTa, DistilBERT, and MPNet—alongside eight specialized evaluation datasets and an interactive demo covering sentence similarity, search, and gender bias.
Fine tuning CLIP with Remote Sensing (Satellite) images and captions
Pre-trained vision-language models like CLIP generalize well across everyday imagery but struggle with remote sensing domain characteristics. To enable text-based search across satellite imagery, a distributed team fine-tuned the openai/clip-vit-base-patch32 model on TPU hardware using JAX and Flax during the Hugging Face Flax/JAX Community Week. The training pipeline incorporated satellite image and caption pairs from the RSICD, UCM, and Sydney datasets alongside extensive image transforms and backtranslation-based text augmentation. Contrastive optimization using Adam with a learning rate of 5e-6 achieved top-1 retrieval accuracy of 0.883 compared to 0.572 on the baseline model. An interactive demo uses a NMSLib index for text-to-image matching, image-to-image matching, and patch-level feature detection.
Sentence Transformers in the Hugging Face Hub
Sentence Transformers has collaborated with the Hugging Face Hub as part of its v2 release to make sentence, paragraph, and image embedding models readily accessible. The Hub now hosts over 90 pretrained models supporting more than 100 languages for use cases like semantic search and multilingual zero-shot classification. Alongside direct model loading in minimal code, the platform provides interactive Hub widgets for feature extraction and sentence similarity computation. Developers can execute inference programmatically through Inference API endpoints. Additionally, the library supports direct model uploading to the Hub via save_to_hub, which automatically generates a model card detailing layer architecture and dual usage with Sentence Transformers and Transformers.
Updating Illustrations at Scale
Shopify’s Polaris team needed to update an illustration style across a product whose assets had become fragmented across more than 6,000 repositories. Illustrators and developers could not reliably identify an image’s repository, code references, reuse, or file type, while GitHub API limits and search behavior prevented dependable large-scale discovery. An initial Node.js monorepo cloned repositories, used globby to find PNG, SVG, and WebP files, searched source lines, and served JSON results through a Preact interface, but rebuilding 24 repositories took more than 35 minutes and produced inaccurate data. The team replaced it with @shopify/get-repo-images, a Go executable that parallelizes file operations, bundles crawling and site generation, and exposes a searchable Next.js interface through NPM. Search and build time fell to eight minutes, with repository cloning as the remaining bottleneck, and the tool was open-sourced.
Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms
Shopify’s Help Center serves more than 2 million users monthly, making search relevance important across setup, marketing, and troubleshooting queries. The post presents a three-step framework for evaluating new search algorithms against an existing system, combining data collection, offline metrics, and online A/B testing. It contrasts Vanilla Pagerank, which boosts articles by total views across searches, with Query-specific Pagerank, which uses click frequencies from historic searches containing the search term. Kafka event streams are modeled into search facts for near-real-time monitoring and experiment assignment, while Support-team annotations provide relevance labels for offline evaluation, with dataset staleness requiring recurring updates. In the reported A/B test, Query-specific Pagerank users clicked more, relied less on follow-up searches and later pages, and reached a lower average clicked rank; the algorithm replaced Vanilla Pagerank.
The GrabMart Journey
During the COVID-19 pandemic, Grab accelerated the rollout of its GrabMart on-demand grocery delivery service to meet surging regional demand. Early pilot testing within the existing GrabFood interface revealed that grocery shopping fundamentally diverges from meal ordering, as users search at an item level rather than browsing merchants. To support grocery workflows, the team restructured the user experience with item and store categories, one-click add-to-cart functionality, and scheduled delivery options. Engineering teams simultaneously updated core infrastructure across order management, driver allocation systems, and merchant Partner APIs to handle specialized mart bookings. Rollouts were validated across two main phases using the internal Grab Early Access program to catch issues before regional release.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.