---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [Predicting Rider Conversion in Sparse Data Environments with Bayesian Trees](https://yomu.fyi/post/predicting-rider-conversion-in-sparse-data-environments-with-bayesian.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Zammit Alban
- Published: Mar 30, 2026
- Type: Problem & solution

Lyft requires real-time predictions of whether a rider will request a ride after viewing price and ETA details to balance supply and demand and personalize user experiences. Traditional machine learning models overfit on high-cardinality, sparse context intersections, whereas deep neural networks introduce unacceptable inference latency during live user sessions. To address this, engineers created a hierarchical Bayesian tree framework that organizes session data into increasingly granular context partitions. Each node houses a parametric model trained top-down, applying Gaussian Bayesian priors and L2 penalties relative to parent parameters to smooth predictions in data-rare child segments. Using simple parametric models at each node also permits monotonic constraints, guaranteeing logically consistent predictions aligned with domain expectations.


### [How Padlet uses ClickHouse Cloud to power real-time classroom analytics](https://yomu.fyi/post/how-padlet-uses-clickhouse-cloud-to-power-real-time-classroom-analytic.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 30, 2026
- Type: Problem & solution

Padlet needed a real-time analytics system to provide teachers with immediate insights into student engagement across 40 million monthly unique users. To avoid the operational overhead of managing database clusters with a small engineering team, the company adopted ClickHouse Cloud and ClickPipes for managed data ingestion. Active user sessions emit 30-second heartbeat events that stream alongside page views into ClickHouse. Materialized views pre-aggregate engagement duration and view counts by date, while native HyperLogLog sketches process unique visitor calculations efficiently across arbitrary time ranges. In production, this architecture ingested roughly eight billion events in a single month and delivered analytical query results with a 45-millisecond median latency.


### [How we’re building a data platform for a new user: agents](https://yomu.fyi/post/how-we-re-building-a-data-platform-for-a-new-user-agents.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Al Brown
- Published: Mar 27, 2026
- Type: Explainer

AI agents are emerging as autonomous database users that generate SQL, execute queries, and iterate in seconds. Because traditional analytics workflows built for human analysts cannot keep up with these rapid loops, ClickHouse is adapting its platform for agentic workloads. The proposed Agentic Data Stack combines LibreChat for conversational interfaces, ClickHouse with Model Context Protocol servers for the data layer, and Langfuse for observability. Supporting developer tooling includes chDB 4 with a Pandas-like API, IDE integrations, and ClickHouse Agent Skills to enforce database design best practices. Platforms like Anthropic and Character.AI already leverage ClickHouse's high-concurrency aggregation and real-time ingestion capabilities to handle massive data scales required by modern AI applications.


### [Building Biz Ask Anything: From Prototype to Product](https://yomu.fyi/post/building-biz-ask-anything-from-prototype-to-product.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Maria Christoforaki, Group Tech Lead; Shree Shalini Pusapati, Software Engineer
- Published: Mar 27, 2026
- Type: Problem & solution

Yelp expanded Yelp Assistant to business pages by developing a question-answering architecture that extracts concise, evidence-backed answers from reviews, photos, and structured facts. The production system replaces static snapshots with near-real-time streaming pipelines for reviews and structured properties alongside weekly batch ingestion for menus and website data. For each incoming query, the system evaluates intent, retrieves relevant context through keyword and embedding searches, and constructs targeted prompts with dynamically selected few-shot examples. Synthesized answers are augmented with supporting visuals and citations before being streamed token-by-token to users. Generating suggested questions directly from specific business content rather than generic categories improved user engagement by roughly fifty percent and reduced unanswerable queries by twenty-six percent.


### [Navigating Regional Network Blocks](https://yomu.fyi/post/navigating-regional-network-blocks.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: sara\_read
- Published: Mar 26, 2026
- Type: Incident / postmortem

Supabase describes how regional network blocks disrupted access to its multi-tenant developer platform and shares three incidents from September 2025 through March 2026. It explains that DNS-level or transport-layer blocking can affect every tenant when projects share domains and IP addresses, while adding a hosting domain to the Public Suffix List provides only partial mitigation. In the UAE, a project distributing unauthorized streaming content triggered an approximately 18-day block that ended after removal and engagement with the TDRA. Yemen saw TCP connections dropped for about eight days with no official explanation, while India’s roughly eight-day restriction was rescinded after project enforcement and technical engagement with MeitY. The post says providers should build regulator contacts, strengthen abuse response pipelines, and develop alternatives such as proxy domains, custom domains, and other workarounds.


### [Top 10 best practices tips for ClickHouse](https://yomu.fyi/post/top-10-best-practices-tips-for-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Yonatan Dolan
- Published: Mar 26, 2026
- Type: Explainer

ClickHouse delivers rapid analytical queries across massive datasets, but suboptimal schemas and queries can significantly reduce performance and storage efficiency. Aligning table definitions with ClickHouse storage and execution mechanics resolves these bottlenecks. Key practices include placing low-cardinality columns first in the ORDER BY clause to enable granule pruning, and selecting optimal data types while avoiding Nullable wrappers to cut memory overhead. Developers should treat partitioning as a data management tool for TTL drops or merge engines rather than a generic speed booster, preventing fragmented parts from slowing query execution. Recent improvements like the Analyzer query planner have also made joins viable, though dictionaries remain faster for static lookups.


### [Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces](https://yomu.fyi/post/beyond-a-b-testing-using-surrogacy-and-region-splits-to-measure-long-t.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Iraklikhorguani
- Published: Mar 25, 2026
- Type: Explainer

Lyft describes a framework for estimating long-term effects of pricing and incentive decisions in its two-sided marketplace, where direct and market-mediated consequences can diverge. The method first uses residualized observational regressions to estimate how policy changes alter distributions of negative experiences such as wait times, surge, cancellations, idleness, and hourly earnings, validating those responses with switch-back experiments. It then applies Augmented Inverse Probability Weighting to estimate how these experiences affect future outcomes, summarizes the relationship with a surrogacy index, and checks calibration using user-split experiments. Finally, Lyft combines mediated and separately estimated direct effects and uses region-split experiments for end-to-end validation, selecting treated and control regions with a forward-selection algorithm inspired by FDiD. The framework is intended to support scenario planning, budget allocation, and continuous calibration of long-run marketplace forecasts.


### [ClickHouse is data lake ready](https://yomu.fyi/post/clickhouse-is-data-lake-ready.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Karolina Ruiz Rogelj, Melvyn Peignon
- Published: Mar 25, 2026
- Type: Announcement

ClickHouse announced expanded data lake capabilities, enabling direct querying, accelerated ingestion into MergeTree, and writes across Apache Iceberg and Delta Lake tables. Open lake architectures provide storage portability across cloud providers and catalogs, but standard query engines often struggle with latency for real-time workloads. Over two years of development, ClickHouse engineered a native Parquet reader with page-level parallelism, integrated the Delta Rust Kernel, and introduced support for schema evolution, time travel, and DML operations. The platform functions across AWS Glue, Unity Catalog, Polaris, and Iceberg REST Catalog on storage systems including Amazon S3, Google Cloud Storage, and Azure Blob Storage. These enhancements allow organizations to execute federated, high-concurrency SQL queries across multiple catalogs or write processed analytical results back to their lakes without vendor lock-in.


### [Intelligent security at ClickHouse speed: How Cogent Security built an AI-native vulnerability management platform](https://yomu.fyi/post/intelligent-security-at-clickhouse-speed-how-cogent-security-built-an.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 24, 2026
- Type: Problem & solution

Enterprise security defenders face shrinking exploitation windows alongside massive volumes of vulnerability findings across disconnected tools. To overcome query bottlenecks from traditional relational databases, Cogent Security designed a lakehouse serving architecture using Apache Iceberg and per-customer ClickHouse instances. The architecture relies on query-driven denormalized tables, custom compression codecs, and pre-computed projections to serve hundreds of millions of rows with sub-second latency. On top of this data layer, Cogent implemented an agentic text-to-SQL system that iteratively queries live data, references an in-house Ontology Service for semantic context, and applies majority-voted evaluations. This combination improved text-to-SQL accuracy from 40% to 94% while providing sub-three-second response times across 98.7% of queries at 100 million rows.


### [Smarter Auto-Scaling for ClickHouse: The Two-Window Approach](https://yomu.fyi/post/smarter-auto-scaling-for-clickhouse-the-two-window-approach.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Ashwath Singh, Manas Alekar
- Published: Mar 24, 2026
- Type: Problem & solution

ClickHouse Cloud required a vertical auto-scaling mechanism that preserves rapid scale-ups while avoiding prolonged over-provisioning after traffic drops. Operating on a single 30-hour lookback window forced scale-downs to take up to 30 hours, whereas an earlier fixed-factor CPU algorithm caused severe oscillations. The engineering team deployed a dual-window recommender pairing a 3-hour window for responsive scale-downs with a 30-hour window for single-step scale-ups. Alongside a trend-based hunting check to resolve conflicting window recommendations, the team introduced target-tracking CPU scaling using geometric mean watermarks and 10-minute median smoothing. This dual-window architecture reduced scale-down latency to 3 hours, prevented scaling oscillations, and significantly reduced infrastructure costs for variable database workloads.


### [Building high-performance full-text search for object storage](https://yomu.fyi/post/building-high-performance-full-text-search-for-object-storage.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Elmi Ahmadov, Jimmy Aguilar, and George Larionov
- Published: Mar 24, 2026
- Type: Problem & solution

ClickHouse redesigned its native full-text index to address latency bottlenecks caused by random read patterns on remote object storage. Instead of using a Finite State Transducer dictionary, the new architecture organizes indexed tokens into fixed-size sorted blocks compressed with front-coding. Each data part separates index data across a small in-memory sparse index file, a sequential dictionary file, and a posting list file. This layout allows the query engine to pinpoint token locations with minimal random I/O and resolve multiple full-text functions directly from index files. Consequently, ClickHouse Cloud maintains predictable search latency across shared object storage while matching local disk performance and scaling through parallel query execution.


### [Making Ads Count: Using MMoE and Auxiliary Tasks to Better Connect Buyers & Sellers](https://yomu.fyi/post/making-ads-count-using-mmoe-and-auxiliary-tasks-to-better-connect-buye.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Amanda Steigman
- Published: Mar 20, 2026
- Type: Explainer

Etsy’s Ads Search ranking model needed a stronger purchase-intent signal because clicks are noisy and purchases are sparse later in the buyer journey. It retained its shared feature-representation and interaction layers, then replaced the shared feed-forward network with Multigate Mixture of Experts (MMoE), using task-specific gates and towers for click-through rate (CTR) and post-click conversion rate (PCCVR). Add-to-cart was added as an auxiliary task, while favorites were rejected after producing negative transfer. Offline, the resulting model increased Purchase and Click PR AUCs by average 3.5% and 1%, respectively, and Purchase ROC AUC by 0.5%. Online, it improved purchases and purchase calibration, supplied better inputs to auto-bidding, kept model size flat through pruning, and reduced inference cost.


### [How Slack Rebuilt Notifications 📣](https://yomu.fyi/post/how-slack-rebuilt-notifications.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Frances Coronel
- Published: Mar 19, 2026
- Type: Problem & solution

Slack redesigned its legacy notification architecture to address user confusion, conflicting desktop and mobile preference systems, and tight coupling between notification content and push delivery. The engineering team unified four separate preference models into a single framework with three channel options: All new posts, Mentions, and Mute. To ensure safe rollback and backwards compatibility without direct database mutations, Slack implemented a read-time translation strategy alongside a new desktop\_push\_enabled preference and auto-saving cross-platform React UI components. This decoupling of activity from push delivery resulted in a fivefold sustained increase in settings engagement, reduced reliance on per-channel overrides, and eliminated synchronization mismatches across mobile and desktop clients.


### [Building a Magic Mirror: AI retail experiences with Remix](https://yomu.fyi/post/building-a-magic-mirror-ai-retail-experiences-with-remix.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 19, 2026
- Type: Tutorial

Shopify’s “magic mirror” is an AI-powered retail installation that uses a display and hidden webcam to deliver personalized messages, animations, and product interactions in under a minute. A Remix route handles the browser and server boundary: a button-triggered POST captures an image, sends it with a curated prompt to a vision-capable language model, and renders a compliment plus a makeup recommendation; server-sent events optionally let a remote device trigger or reset the experience. The prompt combines persona and safety constraints, a plain-text product and inventory table, and a required JSON schema, while strict product-ID lookup provides graceful degradation. The interface uses explicit states, delayed capture, timed response and reset screens, and animated transitions. The production setup keeps brand configuration, shared components, actions, and SSE infrastructure reusable across mirror modes.


### [How ClickStack makes ClickHouse faster for observability](https://yomu.fyi/post/how-clickstack-makes-clickhouse-faster-for-observability.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mike Shi
- Published: Mar 18, 2026
- Type: Problem & solution

ClickHouse delivers high performance for telemetry data, but unoptimized queries on high-cardinality observability workloads can bypass pruning, inflate intermediate state, and exhaust cluster resources. ClickStack addresses this challenge by embedding engine-specific optimization best practices directly into how queries are generated, structured, and executed. For search workflows, the platform executes progressive queries across bounded time windows using optimize\_read\_in\_order to return initial rows rapidly without scanning entire historical ranges. ClickStack also segments large chart aggregations into parallel bucket-aligned queries, automatically redirects map attribute filters to pre-materialized columns, and extends lazy materialization thresholds to minimize runtime input-output overhead. These query formulation strategies are being packaged into specialized analytical endpoints to power notebooks and external tools without requiring manual SQL tuning or exposing raw database interfaces.


### [How Socialpruf built a faster, more reliable data stack by replacing Neon with Postgres managed by ClickHouse](https://yomu.fyi/post/how-socialpruf-built-a-faster-more-reliable-data-stack-by-replacing-ne.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 17, 2026
- Type: Problem & solution

Socialpruf, a social media performance tracking platform ingesting hundreds of posts per second, experienced analytics latency, connection dropouts, and high network transfer costs while running Postgres on Neon alongside ClickHouse Cloud. To resolve these issues, the team migrated their transactional system of record to NVMe-backed Postgres managed by ClickHouse, colocating transactional and analytical workloads. Although standard Postgres logical replication failed on their 0.5 TB dataset, they achieved continuous synchronization using PeerDB before conducting a production cutover. The migration eliminated connection dropouts, reduced network transfer fees, and improved overall Postgres query performance by approximately 30 percent. To manage thousands of concurrent connections generated by ingestion workers, Socialpruf implemented ClickHouse's bundled, horizontally scalable PgBouncer instances.


### [AI is redrawing the database market](https://yomu.fyi/post/ai-is-redrawing-the-database-market.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tanya Bragin
- Published: Mar 17, 2026
- Type: Explainer

Emerging artificial intelligence workloads are fundamentally transforming data platform requirements across real-time analytics, data warehousing, and observability. Conversational analytics and autonomous agents replace periodic batch queries with rapid bursts of concurrent, low-latency requests, exposing the architectural limits of traditional cloud data warehouses and standalone transactional databases. To resolve these constraints, architectures are converging on engine-level integrations between Postgres for transactions and ClickHouse for analytical scale, while unifying operational observability and business event storage. ClickHouse also incorporates turnkey agent interfaces through its acquisition of LibreChat alongside dedicated execution tracing from tools like Langfuse. These combined capabilities ensure organizations can reliably support interactive agentic workflows without incurring severe latency penalties or compounding infrastructure costs.


### [Designing the new async-native ClickHouse Python client](https://yomu.fyi/post/designing-the-new-async-native-clickhouse-python-client.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Joe Spadola
- Published: Mar 16, 2026
- Type: Problem & solution

The official ClickHouse Python client, clickhouse-connect, originally addressed asynchronous execution by wrapping its synchronous HTTP client in a thread pool executor. However, this executor model experienced thread exhaustion, GIL contention, and severe tail latency variance under heavy concurrent workloads. To resolve these scalability bottlenecks, the developers redesigned the client around a half-sync/half-async pattern utilizing aiohttp for async network I/O. A custom bounded queue bridges streaming socket reads on the event loop with synchronous, CPU-intensive binary parsing running inside background threads. Benchmarks demonstrate that this architecture stabilizes tail latencies, delivering an average P95 latency of 556 milliseconds compared to 869 milliseconds for the legacy executor client.


### [AI doesn’t always generate perfect ClickHouse schemas (yet)](https://yomu.fyi/post/ai-doesn-t-always-generate-perfect-clickhouse-schemas-yet.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Al Brown
- Published: Mar 13, 2026
- Type: Explainer

LLMs can generate syntactically valid ClickHouse schemas whose advanced choices are poorly matched to a workload. The post examines recurring pitfalls including query-driven partitioning, per-column codecs, premature projections, materialized-view sprawl, JSON configurations, and forced deduplication with OPTIMIZE TABLE ... FINAL. It explains that partitions primarily support data management, incremental materialized views add work to every insert, and ReplacingMergeTree deduplicates during background merges rather than at insertion. The recommended method is to begin with a simple MergeTree or ReplacingMergeTree, a well-chosen ORDER BY, default compression, and no optional structures, then measure production behavior before adding complexity or seeking human expertise for high-scale, interconnected designs.


### [Shopify’s journey to faster breadth-first GraphQL execution](https://yomu.fyi/post/shopify-s-journey-to-faster-breadth-first-graphql-execution.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 12, 2026
- Type: Explainer

Shopify describes how high-cardinality, deeply nested GraphQL queries exposed execution overhead beyond data loading, with field resolvers assembling responses consuming much of request time. Its GraphQL Cardinal engine replaces conventional depth-first traversal with breadth-first execution, resolving each selected field once across an aggregated set of objects and reducing repeated field-level work. For large list queries, Shopify reports up to 15x faster execution, 90% less memory use, and reductions of many seconds in P50 latency. The migration included a tuned interpreter, cheaper tracer behavior, shadow verification, benchmarks, and ongoing conversion of tens of thousands of legacy resolvers, while future work includes asynchronous patterns and lower-level C bindings.


[Newer posts](https://yomu.fyi/topic/architecture/page/22.md) · [Older posts](https://yomu.fyi/topic/architecture/page/24.md)