# Search > 17 posts about Search, summarised, each linking to the original. ## Articles ### [Say it once: introducing Bot Preference Sync](https://yomu.fyi/post/say-it-once-introducing-bot-preference-sync.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Jin-Hee Lee - Published: Aug 21, 2026 Cloudflare announced Bot Preference Sync, a feature across all customer tiers that automatically reflects dashboard AI bot configurations directly in a site's robots.txt file. Website operators previously faced challenges synchronizing static robots.txt directives with edge enforcement rules, which led some crawlers to ignore stated preferences or attempt bypasses. Under this system, Cloudflare generates and prepends directives for Search, Agent, and Training categories to existing robots.txt files using crawler lists tracked in BotBase. Mixed-use crawlers performing both search indexing and training must meet strict transparency criteria, including honoring opt-outs and providing page-level visibility, to avoid edge blocks when training is disallowed. Additionally, onboarding workflows allow ad-supported publishers to automatically set AI model training to Disallow by default while retaining search visibility. ### [Cloudflare AI Search: give your agents a search engine for your data](https://yomu.fyi/post/cloudflare-ai-search-give-your-agents-a-search-engine-for-your-data.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Nelson Duarte - Published: Aug 6, 2026 Cloudflare introduced developer experience improvements and a preview pricing model for Cloudflare AI Search. Previously, users had to manually connect separate Cloudflare primitives like Workers AI, AI Gateway, Vectorize, R2, and Browser Run to build a search solution. The platform now automates crawling, ingestion, embedding, and retrieval across structured and unstructured data sources. Developers can index sites without sitemaps using link discovery, expose public /search and /mcp endpoints across namespaces, bind search instances directly to Workers, or secure endpoints with Cloudflare Access. Cloudflare uses this system for developer documentation, its corporate blog, and the Cloudflare Dev Stack MCP server to supply coding agents with cited documentation. ### [From ranking to recommended: get your site ready to thrive in the age of AI agents](https://yomu.fyi/post/from-ranking-to-recommended-get-your-site-ready-to-thrive-in-the-age-o.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Matthew Conroy - Published: Aug 6, 2026 Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers. ### [Don’t stop early: Case-folding source code at memory speed](https://yomu.fyi/post/don-t-stop-early-case-folding-source-code-at-memory-speed.md) - Company: [Github](https://yomu.fyi/company/github.md) - Author: Alexander Neubeck - Published: Jul 31, 2026 GitHub's code search engine, Blackbird, must case-fold over 480TB of source code across 180 million repositories during indexing and query matching. To accelerate this operation on source code that is overwhelmingly ASCII, the engineering team replaced early-exit branching with an unconditional branch-free loop. The implementation tests uppercase ASCII ranges using wrapping arithmetic, modifies bits in place, and detects non-ASCII bytes with an accumulator register tested only after the loop completes. Eliminating data-dependent exits allowed LLVM to generate SIMD instructions and achieve throughput exceeding 45 GiB/s on an Apple M4 processor. The optimized implementation was released as the open-source Rust crate casefold. ### [Crowdsourced taxonomy verification: A feedback-driven framework for refining knowledge graph relationships via online search interactions](https://yomu.fyi/post/crowdsourced-taxonomy-verification-a-feedback-driven-framework-for-ref.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Junpeng Niu - Published: Jul 30, 2026 Maintaining accurate Knowledge Graphs in dynamic domains like e-commerce and food delivery is challenging because automated language models frequently hallucinate relationships while manual curation cannot scale. To validate structural taxonomy links continuously, a closed-loop verification framework operationalizes search interfaces by injecting unverified candidate edges as hypotheses into live user traffic. The system uses an exploration-exploitation strategy to place candidate relationships in lower-risk interface slots, tracking contextually anchored micro-interactions such as clicks, dwell times, and purchases. An offline verification engine aggregates these weighted interactions into normalized confidence scores, automatically promoting verified links to permanent graph edges and pruning refuted relationships. ### [Personalizing Airbnb search by learning from the guest journey](https://yomu.fyi/post/personalizing-airbnb-search-by-learning-from-the-guest-journey.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Daochen Zha - Published: Jul 21, 2026 Airbnb replaced hundreds of hand-crafted ranking features with a Transformer-based sequence model that captures both long-term booking history and short-term browsing behavior. To keep latency low and throughput high, sequence representations are generated via daily batch jobs and combined with real-time queries using a co-trained setwise ranker. ### [How Dropbox uses MCP and Dash to close the design-to-code security gap](https://yomu.fyi/post/how-dropbox-uses-mcp-and-dash-to-close-the-design-to-code-security-gap.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra - Published: Jun 12, 2026 Dropbox developed a system using the Model Context Protocol (MCP) and Dash's semantic search to bridge the gap between security threat models and code implementation. By retrieving original security documents during pull requests, an LLM agent automatically evaluates whether the proposed code adheres to previously agreed-upon security requirements. This approach surfaces design regressions and missing controls that traditional static analysis tools miss. ### [From decentralized Docs-as-Code to a centralized repository: Evolving Grab's documentation strategy](https://yomu.fyi/post/from-decentralized-docs-as-code-to-a-centralized-repository-evolving-g.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Karen Kue - Published: May 29, 2026 Grab originally adopted a decentralized Docs-as-Code workflow to manage Markdown documentation alongside service code, but the model strained as engineering scaled. Dispersed repositories led to fragmented terminology, uncoordinated repository moves, uneven linting standards, and noisy search results. To resolve these issues, the organization transitioned to a centralized documentation repository with standardized templates, automated CI quality gates, and a unified search overlay combining fuzzy title search with Glean. Consolidating Markdown into a single repository also established a clean, structured knowledge base tailored for internal AI agents. The centralized setup fostered continuous maintenance, averaging 27 daily commits and an update merged roughly every 50 minutes. ### [How we optimized Dash's relevance judge with DSPy](https://yomu.fyi/post/how-we-optimized-dash-s-relevance-judge-with-dspy.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon - Published: Mar 17, 2026 Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling. ### [Using LLMs to amplify human labeling and improve Dash search relevance](https://yomu.fyi/post/using-llms-to-amplify-human-labeling-and-improve-dash-search-relevance.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon,Dmitriy Meyerzon - Published: Feb 26, 2026 Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time. ### [LLM-assisted vector similarity search](https://yomu.fyi/post/llm-assisted-vector-similarity-search.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Md Riyadh - Published: Oct 23, 2024 Vector similarity search often struggles with conceptual nuances, negations, and complex constraints despite its efficiency in high-dimensional semantic retrieval. To address this limitation, a two-step retrieval pipeline combines Facebook AI Similarity Search (FAISS) using OpenAI's text-embedding-ada-002 embeddings with a large language model re-ranking step powered by GPT-4o. The initial FAISS search extracts a candidate shortlist of matches, which GPT-4o then re-ranks based on contextual relevance and query intent. Evaluations on structured datasets showed that while simple queries yielded comparable results across methods, the LLM-assisted approach effectively resolved negations and conceptual matching where raw vector search failed. Internal production trials on relational datasets exceeding 4,500 rows demonstrated similar relevance improvements, with additional query latency being the primary operational trade-off. ### [Enabling conversational data discovery with LLMs at Grab](https://yomu.fyi/post/enabling-conversational-data-discovery-with-llms-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shreyas Parbat - Published: Sep 26, 2024 Grab faced major bottlenecks in dataset discovery across its 200,000 data lake tables due to low documentation coverage and vanilla Elasticsearch limitations in its internal tool, Hubble. To resolve this, engineers first tuned Elasticsearch by boosting frequently queried P80 tables and certified schemas while deboosting deprecated datasets. They then deployed a GPT-4 pipeline that generated documentation from table schemas and sample data, raising P80 documentation coverage from 20% to roughly 90%. Finally, the team developed HubbleIQ, an LLM-powered assistant built via Glean Apps and integrated into Slack and Hubble search to answer semantic queries. These initiatives increased search click-through rates to 94% and lifted the proportion of users finding discovery easy to 73%. ### [Query expansion based on user behaviour](https://yomu.fyi/post/query-expansion-based-on-user-behaviour.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shuailong Liang - Published: Nov 16, 2022 Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone. ### [How facial recognition technology keeps you safe](https://yomu.fyi/post/how-facial-recognition-technology-keeps-you-safe.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Kai Feng Tee - Published: Jun 9, 2022 Grab utilizes facial recognition technology across its platform for driver authentication, passenger verification, and digital electronic Know Your Customer (e-KYC) processes. The core pipeline consists of image preprocessing through face detection and alignment, anti-spoofing checks, feature extraction into high-dimensional vector embeddings, and downstream verification or search. To counter spoof attacks like screen replays, synthetic moiré patterns are generated and cropped face patches are used during training and inference to focus on local structures rather than global semantic noise. Face verification challenges involving shallow ID datasets and masked faces are resolved using semi-Siamese training architectures and masked data augmentations. ### [The GrabMart Journey](https://yomu.fyi/post/the-grabmart-journey.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Clarisse Peralta - Published: Jan 18, 2021 During the COVID-19 pandemic, Grab accelerated the rollout of its GrabMart on-demand grocery delivery service to meet surging regional demand. Early pilot testing within the existing GrabFood interface revealed that grocery shopping fundamentally diverges from meal ordering, as users search at an item level rather than browsing merchants. To support grocery workflows, the team restructured the user experience with item and store categories, one-click add-to-cart functionality, and scheduled delivery options. Engineering teams simultaneously updated core infrastructure across order management, driver allocation systems, and merchant Partner APIs to handle specialized mart bookings. Rollouts were validated across two main phases using the internal Grab Early Access program to catch issues before regional release. ### [Pharos - Searching Nearby Drivers on Road Network at Scale](https://yomu.fyi/post/pharos-searching-nearby-drivers-on-road-network-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Hao Wu - Published: Dec 22, 2020 Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups. ### [Save Your Place with Grab!](https://yomu.fyi/post/save-your-place-with-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Summit Saurav - Published: Aug 1, 2019 Grab introduced Saved Places across Southeast Asia to eliminate the friction of repeatedly typing addresses and prevent selection errors between similarly named locations. Data analysis across transport and food orders revealed that consumers consistently visit only five to seven unique locations and order food to one or two addresses. To streamline repeat bookings, cross-functional teams designed a feature allowing users to bookmark locations under custom labels such as Home and Work. Following usability testing and release, more than 14 million users stored nearly 45 million addresses across the platform. Platform metrics also showed that while office destinations clustered in central districts across major cities, residential distributions varied significantly between markets like Singapore and Jakarta.