Bloom Filters: The Cheap Gate Before the Expensive Question
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
Loading…
Topic
Search
181 posts about Search. Every summary links to the original.
- 181 posts
- 31 companies
- 3 explainers
See the idea
Search explainers
Apache Lucene Scoring: BM25 Relevance vs StableTfl Consistency
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
AI Search: Speed vs. Relevance Trade-off
AI Search balances retrieval speed and result quality by choosing between dense embeddings, sparse lexical search, and hybrid approaches. Understand which trade-off fits your workload.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Boolean Query Operators Guide, With Examples
Social listening queries filter relevant brand, product, industry, or consumer mentions from large volumes of online conversation, beginning with the research questions the analysis should answer. The guide explains Boolean operators including quotation marks for exact phrases, OR for alternatives, AND for context, NEAR/x for proximity, NOT for exclusions, wildcards for variations, and brackets for grouping. It recommends adding misspellings, hashtags, account handles, websites, and exclusions, then testing and adjusting the query while keeping its structure simple where possible. Brandwatch Consumer Research offers Boolean, non-Boolean, and blank query options, with editor validation and Iris AI suggestions for entities, keywords, hashtags, subreddits, and multilingual terms.
Image replacement in Canva designs using reverse image search
Canva needed an automated way to replace media in design templates, such as when third-party licensing partnerships expire across more than 150 million images. Existing recommendation engines, perceptual hashing, and text metadata searches failed to capture visual similarity hierarchies or ensure replacement relevance. To build a reverse image search system, engineers evaluated embedding models including CLIP, ViTMAE, DreamSim, CaiT, and DINOv2 alongside an external vector database supporting metadata filtering. Evaluation on sample datasets identified DINOv2 as the best model for preserving subjects, background context, and color tones in photos. Integrated into the Template Assistant as a human-in-the-loop tool, the automated suggestions increased image replacement speeds by 4.5 times during initial pilot testing.
LLM-assisted vector similarity search
Vector similarity search can struggle with queries containing negation, constraints, or conceptual requirements, even though it efficiently retrieves semantically similar records. The proposed method uses FAISS and OpenAI text-embedding-ada-002 embeddings to shortlist 15 candidates, then asks GPT-4o to rank the candidates against the original query and return the top three. On two GPT-4o-curated synthetic datasets—100 food dishes and 100 Asian tourist spots—the assisted method matched raw search on straightforward queries but produced more relevant results for complex ones, including excluding fish or shrimp and identifying wildlife-related places. The authors note limits from the small datasets and query set, report similar relevance improvements in an internal deployment involving more than 4,500 relational-database rows, and identify shortlist size and added LLM latency as areas for further evaluation.
How Shopify improved consumer search intent with real-time ML
Shopify describes how it built near-real-time text and image embedding pipelines to support semantic search, replacing keyword matching with representations that enable similarity-based retrieval. Using Google Cloud’s Dataflow and Apache Beam, the pipelines process roughly 2,500 embeddings per second, preprocess images, run GPU-backed inference, and write outputs to both a data warehouse and an event topic for downstream ingestion. The engineering discussion focuses on trade-offs among memory, throughput, latency, and cost: reducing harness threads cut memory use about 2.6x and allowed a return to cheaper workers, while model sharing reduced throughput. For batching, bursty input produced bundles of one, and stateful batching was rejected because its shuffle increased latency; Dataflow’s per-process model instances instead provided enough parallelism to saturate GPUs.
What is a Vector Database?
Vector databases store and manage high-dimensional vector representations of data, grouping them by similarity rather than traditional row-and-column organization. They support semantic and similarity-based retrieval by converting text, images, audio, or video into embeddings, indexing them with methods such as product quantization and HNSW, and comparing query vectors with indexed vectors. These capabilities underpin use cases including RAG, hybrid search, recommendations, NLP, image and video recognition, and anomaly detection, while sharding, partitioning, caching, and replication support scale. The article distinguishes a vector database from a standalone index because it also provides storage, CRUD, metadata filtering, horizontal scaling, and data management functions, and describes Databricks AI Search as an integrated implementation with automatic scaling, optimization, security, and governance.
Enabling conversational data discovery with LLMs at Grab
Grab faced major bottlenecks in dataset discovery across its 200,000 data lake tables due to low documentation coverage and vanilla Elasticsearch limitations in its internal tool, Hubble. To resolve this, engineers first tuned Elasticsearch by boosting frequently queried P80 tables and certified schemas while deboosting deprecated datasets. They then deployed a GPT-4 pipeline that generated documentation from table schemas and sample data, raising P80 documentation coverage from 20% to roughly 90%. Finally, the team developed HubbleIQ, an LLM-powered assistant built via Glean Apps and integrated into Slack and Hubble search to answer semantic queries. These initiatives increased search click-through rates to 94% and lifted the proportion of users finding discovery easy to 73%.
Improving Retrieval on Ramp with Transaction Embeddings
Ramp developed transaction embeddings to help automate accounting coding, where employees choose general ledger (GL) categories for transactions. The model represents enriched transaction features—such as merchant, category, department, location, amount, memo, spend program, and trip—as stringified documents labeled with chart-of-accounts codes. Starting from a pre-trained encoder, it is fine-tuned with sentence-transformers and BatchSemiHardTripletLoss, using large batches and contextual sampling to learn embeddings whose similarity is measured with cosine similarity. Triplet loss pulls same-label transactions together and pushes different-label examples apart, while informed mining addresses subtle distinctions such as Travel: Sales versus Travel: Engineering. The embeddings support GL coding suggestions, high-confidence defaults, business-spend similarity analysis, and transaction context for LLM-enabled features, with the stated aim of keeping predictions personalized and data private.
eBay Introduces Intuitive Search Redesign to Elevate Shopper Experience
eBay redesigned its search experience after 18 months of user research and testing, targeting a more seamless, intuitive, and visually rich shopping journey. The update uses larger high-resolution listing images, a modernized information layout, redesigned navigation menus, and streamlined filters, with particular attention to visual categories such as Clothing, Shoes & Accessories, Jewelry & Watches, and Toys & Hobbies. A new Shopping View replaces the previous gallery view with a full-width display without sidebar ads, while delivery options are consolidated and the price filter adds an interactive graph showing price distribution. eBay says it will continue using artificial intelligence and machine learning to refine relevance, listing benefits, and seller details, with the stated aim of improving efficiency and trust for buyers and sellers.
From Pre-trained to Fine-tuned: Nextdoor’s Path to Effective Embedding Applications
Nextdoor transitioned its ranking and recommendation pipelines from traditional continuous and discrete interaction features to transformer-based representation learning. The engineering team deployed pre-trained Sentence-BERT models to generate multilingual post and comment representations, which were aggregated daily by interaction type to form user embeddings. To improve search recall, the team fine-tuned sentence transformers on unlabeled query session logs using contrastive learning and integrated HNSWlib for approximate nearest neighbor retrieval. Subsequent iterations incorporated labeled feedback, BERTopic for coarse personalization, and experiments with CLIP image embeddings. Infrastructure scaling challenges were addressed by performing embedding transformations directly within FeatureStore and optimizing feature payload formats to minimize microservice network bandwidth.
pgvector: Fewer dimensions are better
Embeddings support semantic search, retrieval augmented generation, and clustering in pgvector, but high-dimensional vectors increase storage, memory, and distance-computation costs as datasets grow. The post explains how IVF indexes reduce the records examined and compares embedding models using Massive Text Embedding Benchmark results, noting little correlation between dimension count and performance. In a 224,482-vector Wikipedia dataset, all-MiniLM-L6-v2 with 384 dimensions delivered 78% higher queries per second than OpenAI text-embedding-ada-002 with 1536 dimensions at accuracy@10 of 0.99, while using 4GB rather than 7.5GB of RAM. The 384-dimensional gte-small model produced more than a 200% queries-per-second improvement over all-MiniLM-L6-v2 at the same accuracy, though model choice still depends on task performance, sequence length, language support, and operational costs.
pgvector 0.4.0 performance
The post benchmarks pgvector 0.4.0 for approximate nearest-neighbor search and assesses whether PostgreSQL can support production-scale vector workloads. Using the ANN Benchmarks methodology, pre-warmed RAM, the dbpedia-entities-openai-1M dataset of 1 million 1,536-dimensional OpenAI embeddings, and Supabase ARM instances, it varies probes, lists, concurrency, and compute size. On an 8-core, 32 GB instance, pgvector reached 0.91 accuracy@10 at 380 QPS with 10 probes and 0.98 accuracy at 140 QPS with 40 probes; a 64-core, 256 GB server reached about 1,800 QPS at 0.91 accuracy. The results indicate predictable scaling, while higher lists can improve query speed at the cost of index-build time, and the roadmap identifies HNSW, product quantization, and parallel index builds as planned improvements.
Supabase Vecs: a vector client for Postgres
Vecs is a new Python library for managing embeddings in Postgres databases that use pgvector, offering a vector-store-like interface for table creation, indexing, querying, and metadata filtering. Collections are stored in a vecs schema with an id text primary key, a vector column, and JSONB metadata; queries support cosine distance, l2 distance, and maximum inner dot product, with optional filters and returned values. A short Python workflow can create a collection, upsert records, create an index after insertion, and search by vector; the collection can also be queried without an index. With Supabase, a collection can be exposed through a public-schema view and accessed from supabase-js, while proposed preprocessing transforms and an asynchronous client remain under debate.
ChatGPT plugins now support Postgres & Supabase
Retrieval plugins let ChatGPT access relevant content from private datasets by chunking documents, generating embeddings with OpenAI’s text-embedding-ada-002, and querying a vector database. The article describes Postgres and Supabase datastore implementations using pgvector, with a shared documents schema and a match_page_sections function that filters by source, author, document, and date before ranking vectors with inner product distance. It then gives a local example: downloading the Postgres 15 documentation as a PDF, uploading it through the plugin’s /upsert-file endpoint, and configuring environment variables for Supabase or Postgres. The example produced 7,904 document records, and an optional IVFFlat index can trade exact nearest-neighbor search for approximate search; after registering the local plugin in ChatGPT, users can ask questions answered from the documentation.
Launch Week 7 Hackathon Winners
Supabase Launch Week 7 announces the winners of its hackathon, which introduced an AI theme and recognized projects across five categories. Page Assist won Best Overall for an open-source Chrome extension that lets users chat with webpages through the ChatGPT API, while other winners included Rect, Supathreads, Replay.ai, Supa0SQL, and Generation: Hotdog. The projects apply AI to educational content, blog-to-thread conversion, browsing-history search with cosine similarity, natural-language chart creation, and hotdog-image generation using OpenAI and Stable Diffusion APIs; runner-ups included tools for video chat, drum patterns, tweet scheduling, and music guessing. Members of the Best Overall winning team receive a limited-edition Launch Week 7 Keyboard, while winners and runner-ups in other categories receive Launch Week 7 swag kits, alongside guides for getting started with Supabase resources.
dbdev: PostgreSQL Package Manager
database.dev is a public preview of dbdev, a PostgreSQL package manager whose registry is currently read-only and preloaded with packages demonstrating packaging and discovery possibilities. It aims to provide repeatable deployment of libraries and applications, with third-party publishing tools planned after community feedback, and uses organization- or user-namespaced package names to address name and typo squatting. The in-database client installs packages, which PostgreSQL then exposes as native extensions; dbdev adds major.minor.patch version constraints, while compatibility currently requires Supabase new projects or PostgreSQL with pg_tle and pgsql-http. Examples include a hybrid document-search backend, an index advisor that reports a sample total-cost reduction from 25.88 to 6.40, and administration views for database diagnostics. The preview depends on Trusted Language Extensions, and the post invites reports, feature ideas, and contributions while noting that pl/rust had not yet reached Supabase.
Supabase Clippy: ChatGPT for Supabase Docs
Supabase introduced “Supabase Clippy,” a ChatGPT-like interface trained on its documentation, as a hidden MVP in the Supabase docs. It responds to questions by retrieving relevant documentation and streaming GPT-3's answer in realtime. The implementation parses docs into sections, creates OpenAI embeddings, stores them in Postgres with pgvector, and injects matching documents as context for generation. The project was motivated by the growth of Supabase's product suite and documentation, plus the need to support developers unfamiliar with Postgres; the current version uses only Guides and is presented as a one-week MVP intended to gather feedback. Planned directions include adding reference docs and community discussions, caching repeated queries, thumbs-up relevance feedback, and interactive guidance.
Storing OpenAI embeddings in Postgres with pgvector
Post describes how embeddings represent semantic relatedness as vectors and how OpenAI’s text-embedding-ada-002 model produces 1536-dimensional embeddings for text. It addresses storing and searching large or dynamically changing collections that may not fit comfortably in memory, using PostgreSQL and the pgvector extension instead of a small CSV-and-NumPy workflow. The implementation enables the vector extension, creates a documents table with a vector(1536) column, and defines match_documents() with cosine distance, thresholds, counts, and ordering. It also shows JavaScript code for generating embeddings, a search flow that retrieves matching documents for prompt context, and token limiting before a completion request, while noting that documents can be chunked. For larger tables, the post recommends an ivfflat index with vector_cosine_ops and gives 4 * sqrt(table_rows) as a starting point for lists.
Launch Week 6: Wrap Up
Supabase’s Launch Week Day 6 wrap-up consolidates the products and platform changes announced during the event. The roundup announces redesigned Next.js documentation, Storage v2 features for image resizing, webhooks, and a Smart CDN, plus multi-factor authentication usable in Row Level Security policies. It also introduces Wrappers, an Alpha framework for connecting Postgres to external systems, Vault for encrypted secrets, and pg_graphql 1.0 with general availability. Additional releases include dashboard-enabled point-in-time recovery for Pro Projects, custom domains, pg_crdt, PostgreSQL 15 for new projects, a PostgREST 11 pre-release, and the PGroonga multilingual full-text search extension. Community partnerships with Flutterflow and OneSignal provide Flutter app development and push-notification integrations.
Query expansion based on user behaviour
Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.
Postgres Full Text Search vs the rest
The post compares PostgreSQL Full Text Search with dedicated search engines including MeiliSearch, OpenSearch, SQLite FTS, and Typesense, asking whether a separate system is justified for common application search. It presents a PostgreSQL setup using a generated stored column built with to_tsvector('english', ...) and a GIN index, queried with websearch_to_tsquery. The benchmark loads a movie dataset and focuses on query speed after data ingestion, using comparable scripts and default configurations with limited tuning. PostgreSQL was close in performance to several alternatives; in-memory SQLite was fastest, while Typesense and MeiliSearch handled misspellings correctly and OpenSearch ingested data quickly. The conclusion is that PostgreSQL can provide good-enough search quality and speed for basic cases without adding another system, although dedicated engines offer capabilities PostgreSQL does not.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.