Bloom Filters: The Cheap Gate Before the Expensive Question
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
Loading…
Topic
Search
181 posts about Search. Every summary links to the original.
- 181 posts
- 31 companies
- 3 explainers
See the idea
Search explainers
Apache Lucene Scoring: BM25 Relevance vs StableTfl Consistency
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
AI Search: Speed vs. Relevance Trade-off
AI Search balances retrieval speed and result quality by choosing between dense embeddings, sparse lexical search, and hybrid approaches. Understand which trade-off fits your workload.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Say it once: introducing Bot Preference Sync
Cloudflare announced Bot Preference Sync, a feature across all customer tiers that automatically reflects dashboard AI bot configurations directly in a site's robots.txt file. Website operators previously faced challenges synchronizing static robots.txt directives with edge enforcement rules, which led some crawlers to ignore stated preferences or attempt bypasses. Under this system, Cloudflare generates and prepends directives for Search, Agent, and Training categories to existing robots.txt files using crawler lists tracked in BotBase. Mixed-use crawlers performing both search indexing and training must meet strict transparency criteria, including honoring opt-outs and providing page-level visibility, to avoid edge blocks when training is disallowed. Additionally, onboarding workflows allow ad-supported publishers to automatically set AI model training to Disallow by default while retaining search visibility.
Build a unified AI agent architecture with DynamoDB and Bedrock
AI agent architectures on AWS frequently divide operational records and vector embeddings across separate databases, creating synchronization lag and infrastructure overhead. To resolve this fragmentation, a single DynamoDB table can store operational document attributes alongside embeddings by leveraging native vector search. An Amazon Bedrock agent directs incoming queries to AWS Lambda action groups, which run standard CRUD operations or semantic searches against the DynamoDB SearchVectors API. When items change, a DynamoDB Streams pipeline uses Amazon Titan Text Embeddings V2 to generate new embeddings and write them back to the table. This unified approach eliminates standalone vector databases while keeping semantic indexes synchronized with operational updates.
August 2026 newsletter
The August 2026 newsletter rounds up ClickHouse product releases, research, managed services, observability, community work, and upcoming events. It introduces Andy Pavlo’s arrival to establish ClickHouse Labs, describes the 26.7 release’s join optimizations, token-position text indexes, and EXPLAIN ANALYZE, and reports a 40× phrase-search improvement on a Hacker News dataset. It also covers an interactive ClickBench playground for more than 100 database systems, running isolated workloads in Firecracker microVMs on AWS EC2, plus guidance on combining Postgres for mutable data with ClickHouse for immutable analytical workloads. Additional items address Managed Postgres notifications and backups, TTL-related disk growth, vector-search trade-offs, and hdx-evals, which found a purpose-built ClickStack MCP server scored 7–20 percentage points higher than direct SQL access in five reproducible scenarios.
How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code
Papers with Code needs a search engine that supports exact titles and arXiv identifiers, semantic requests, navigational queries, typos, and fast responses when model services are cold or unavailable. The system uses hybrid retrieval: PostgreSQL full-text search provides a lexical baseline, pgvector stores dense embeddings, and reciprocal rank fusion combines them. Full-corpus embeddings for more than 110,000 papers run in burstable GPU Jobs from repeatable-read database snapshots, with checksummed JSONL inputs and resumable Parquet outputs stored in Buckets. An Inference Endpoint serves low-latency query embeddings and bounded hourly updates for changed papers, while lexical retrieval remains a fallback. The embedding contract pins the model revision, dimension, prompts, normalization, and source hash; production uses Qwen/Qwen3-Embedding-0.6B with 256-dimensional L2-normalized vectors, and stored vectors also support related-paper retrieval without a request-time model call.
Building Menu Vision: Real-Time Dish Recognition
Yelp developed Menu Vision to help diners recognize menu items and view corresponding user-generated photos and reviews in real time using their phone cameras. The feature evolved from a two-day hackathon Android prototype into a production system deployed across iOS and Android. Client devices perform all optical character recognition, text cleanup, and dish matching locally using native machine learning frameworks, while dish metadata is prefetched upon opening a restaurant page. Backend pipelines combine partner data, reviews, and photo captions into a Cassandra database, filtering exclusively for dishes with photos. To handle optical character recognition errors and name variations, the matching pipeline runs exact matching, bidirectional substring matching, and Jaro-Winkler similarity matching.
The P-Completeness of Inverted Index Traversal: On the Complexity of Evaluating Boolean Query DAGs
Modern artificial intelligence agents increasingly rely on search infrastructure to run complex, neuro-symbolic reasoning workflows that compile into deeply nested, non-monotonic Boolean queries. Standard inverted index evaluation methods struggle with these query structures, facing either exponential query complexity under Document-at-a-Time models or universe-scale space penalties under Term-at-a-Time models during logical negation. To address these theoretical limitations, the author formalizes a DAG-based retrieval language named L_R and proves its query evaluation problem is strictly P-Complete. The paper introduces ComputePN, a deterministic and sparsity-aware algorithm that employs a Positive-Negative dual representation alongside native DAG memoization. This mechanism bounds evaluation time to O(|Q| · |U_active|), effectively evaluating P-Complete queries natively over the index while bypassing combinatorial tree expansion and universal scan overhead.
Evaluating AI Agents Live at the Grounded Reasoning Cup
The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Sentence Transformers v6.0 adds MultiVectorEncoder, bringing ColBERT-style late-interaction retrieval into the library’s existing API for dense, sparse, and reranker models. The encoder retains one projected, typically 128-dimensional vector per token instead of pooling a document into one vector, allowing documents to be encoded independently and indexed offline. At scoring time, MaxSim finds each query token’s highest dot-product match among document tokens and sums those maxima; contextual embeddings support both paraphrase matching and exact-token evidence. The guide covers checkpoint loading, encoding, scoring, semantic search, retrieval and reranking, indexing, token pooling, inference speed, evaluation, and retrieval from page images, audio, and video. It also quantifies the storage trade-off: 4,874 passages produced 608,414 token vectors and a 311.5 MB float32 index, while fast-plaid compression reduced that example to 92 MB.
Measuring campaign impact with social and search data
Campaign post-mortems often report reach, engagement, impressions, and sentiment without establishing whether a campaign changed perception or behavior. It argues that social listening should measure sentiment, themes, mention volume, and audience conversation shifts, while search intelligence measures category and branded demand, promoted benefits, and share of search. A client example contrasts a campaign that generated strong social buzz while category search declined with a later campaign that reversed declining demand, captured two-thirds of share of search, and grew search intent 134%. The difference was attributed to combining socially engaging creative with media strategy and messaging designed to drive search demand. The proposed framework aligns measurement with campaign goals and uses gaps between social sentiment and search intent to identify awareness without demand, existing intent without audience expansion, or stronger market impact.
What’s new in the ClickHouse Grafana plugin
ClickHouse’s Grafana plugin has shipped a set of changes aimed at making common observability tasks—finding log lines, narrowing to services, and detecting deployments—faster without requiring SQL. Version 4.20 adds single-table configuration and compact query mode, where users search logs, apply type-aware filters, review query history, and see or edit generated ClickHouse SQL. Investigation can proceed by clicking through collapsible OpenTelemetry Resource, Scope, and Log attributes, filtering values or selected message text, while dashboards can discover Map keys and values across tables. The release also includes interactive Logs Explorer, Traces Explorer, and per-service OpenTelemetry dashboards, plus guided variable and annotation editors that generate editable SQL for distinct values and change markers such as deployments and rollbacks. Additional changes cover schema-introspection caching, trace-ID lookup, Grafana 13 compatibility, SQL validation, backend OpenTelemetry instrumentation, and forwarding Grafana identity to ClickHouse; compact mode initially remains limited to single-table data sources.
Introducing OlmoEarth embeddings: Custom embedding exports from OlmoEarth Studio for downstream analysis
OlmoEarth Studio now supports custom computation and export of embedding vectors, compact representations of Earth-observation data generated by open-source OlmoEarth foundation models. Users can select an area, a one-to-twelve-month time span, an encoder variant, spatial resolution, and Sentinel-2 L2A or Sentinel-1 RTC imagery, then receive a Cloud-Optimized GeoTIFF with one int8 band per embedding dimension. The post demonstrates similarity search, few-shot segmentation, change detection, and PCA-based exploration; a logistic regression trained on 60 labeled pixels produced a weighted F1 of 0.84 for three land-cover classes in Ca Mau, Vietnam. Embeddings provide a fast, shareable starting point, while supervised fine-tuning can improve performance for applications needing task-specific results; cloud cover, atmospheric artifacts, and missing observations can affect quality.
What's new in Sentry Logs: The summer 2026 roundup
Sentry introduced a collection of updates to its logging platform aimed at improving search scalability, export flexibility, and general viewer usability. The system now supports URL-synced log pinning alongside an overhauled export interface that handles aggregate data, full attribute payloads, and high-volume JSONL formats across Logs, Errors, and Tracing. Performance enhancements to the log viewer include a more constrained layout with reduced initial data loads, inline error representations, individual chips for multi-value search terms, and trace-level filtering. To assist large-scale queries spanning hundreds of gigabytes or terabytes, frontend and backend modifications added longer query timeouts with continuations, clearer rate-limiting messages, and in-place retries.
Whats new in ClickStack - June + July
ClickStack’s June–July releases add a broader set of observability features, with work centered on trace navigation, metrics sources, dashboards, and filter performance. The trace waterfall now assigns stable colors by service, keeps span durations outside bars, preserves clickable proportions for very short spans, and adds a minimap plus OpenTelemetry span-link navigation. Metrics support includes quantiles for exponential histograms, a leaner default schema, and an experimental UI connection to external Prometheus-compatible endpoints, while PromQL is proxied to the configured endpoint. Dashboard filters can narrow one another, event patterns can appear as tiles, kiosk mode provides a read-only display, and filter-value discovery prefers ClickHouse text indexes when available. Additional changes cover alert annotations, browser RUM dashboard views, external links from table tiles, grid snapping, and service-map coloring by latency, error rate, or throughput.
Which brand won the football tournament footwear battle
An analysis of footwear conversations around the 2026 World Cup compares how sponsorship, search demand, social attention, and sentiment shaped the competition between Adidas, Nike, Puma, New Balance, and other brands. Consumer Research tracked tournament-related mentions from June 11 to July 19 and paired them with search queries across the US, UK, and Canada. Adidas led tournament-specific footwear conversation with 63% of mentions, while Nike led broader brand discussion with 995k mentions, brand search momentum, and 78% positive sentiment. Nike’s cultural activations, including LISA’s “Rip The Script” campaign and Ronaldo’s Mercurial moments, generated shareable attention, whereas Adidas benefited from official sponsorship, the Trionda match ball, and player storytelling. Pink cleats became a cross-brand liability, attracting more than 5.5k mentions with nearly 70% negative sentiment, showing that visibility did not automatically produce favorable attention.
Cloudflare AI Search: give your agents a search engine for your data
Cloudflare introduced developer experience improvements and a preview pricing model for Cloudflare AI Search. Previously, users had to manually connect separate Cloudflare primitives like Workers AI, AI Gateway, Vectorize, R2, and Browser Run to build a search solution. The platform now automates crawling, ingestion, embedding, and retrieval across structured and unstructured data sources. Developers can index sites without sitemaps using link discovery, expose public /search and /mcp endpoints across namespaces, bind search instances directly to Workers, or secure endpoints with Cloudflare Access. Cloudflare uses this system for developer documentation, its corporate blog, and the Cloudflare Dev Stack MCP server to supply coding agents with cited documentation.
From ranking to recommended: get your site ready to thrive in the age of AI agents
Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers.
ClickHouse Release 26.7
ClickHouse 26.7 is a monthly release with 61 new features, 112 performance optimizations, and 329 bug fixes, including improvements to aggregation, joins, vector search, phrase search, query analysis, and URL access. Its combination of optimize_aggregation_in_order and the new optimize_aggregation_in_order_limit turns matching GROUP BY, ORDER BY, and LIMIT queries into low-memory streaming pipelines that can stop reading after N groups. In a TPC-H scale-factor-100 test, the optimized query ran 313× faster, processed 339× fewer rows, and used 592× less peak memory. The release also adds build-side JOIN pruning, LLVM JIT compilation for supported regular expressions, and macOS improvements including faster allocation-heavy workloads, concurrent remote-replica reads, SSD cache dictionaries, FileLog, and STREAM support.
How Physical Intelligence unified its robotics data stack with Postgres managed by ClickHouse
Physical Intelligence, which builds robotics foundation models, needed a data stack that could support both transactional operations and high-cardinality analytical workloads. Its previous single RDS Postgres instance could not handle those demands, so the team migrated to ClickHouse Cloud and adopted Postgres managed by ClickHouse for OLTP alongside ClickHouse for OLAP. Postgres is automatically replicated into ClickHouse through ClickPipes, while materialized views, full-text indices, and a query agent help researchers search across the company’s data. The resulting stack supports faster, easier exploration as data volume grows from the 10 billion to 100 billion row range, and also handles telemetry through ClickHouse with ClickStack and Grafana.
Don’t stop early: Case-folding source code at memory speed
GitHub's code search engine, Blackbird, must case-fold over 480TB of source code across 180 million repositories during indexing and query matching. To accelerate this operation on source code that is overwhelmingly ASCII, the engineering team replaced early-exit branching with an unconditional branch-free loop. The implementation tests uppercase ASCII ranges using wrapping arithmetic, modifies bits in place, and detects non-ASCII bytes with an accumulator register tested only after the loop completes. Eliminating data-dependent exits allowed LLVM to generate SIMD instructions and achieve throughput exceeding 45 GiB/s on an Apple M4 processor. The optimized implementation was released as the open-source Rust crate casefold.
Crowdsourced taxonomy verification: A feedback-driven framework for refining knowledge graph relationships via online search interactions
Dynamic catalogs in food delivery and e-commerce make knowledge-graph (KG) maintenance difficult: LLM- or scraper-generated relationships can be semantically wrong, manual review does not scale, and bad edges can degrade query expansion, CTR, and trust. The proposed feedback-driven verification engine treats unverified edges as hypotheses and uses the search interface as a validation environment. It separates verified and candidate edges, injects a limited number of candidates into lower-risk interface positions through an exploration-versus-exploitation policy, and records context-anchored user interactions with weighted positive and negative signals. An offline engine aggregates exposure and engagement into a confidence ratio, promoting candidates above a verification threshold and pruning those below a failure threshold. The framework is presented as a continuous human-in-the-loop method for validating parent-child and sibling relationships at scale; planned extensions add confidence tiers and defenses against spam, bots, and Sybil attacks.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.