Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
Apache Pinot sits at the serving layer between event pipelines and applications that need fast analytical answers. Kafka and Flink can deliver Schema Registry-compatible metrics, while Pinot uses partitioned, columnar, indexed data to support millisecond queries across large time ranges. Replica groups and pool-based instance assignment help preserve query and ingestion availability during zone failures. The diagram also highlights the Star-tree trade-off: faster aggregation queries can require substantially more storage.
The posts behind it
Clickhouse · · ExplainerClickHouse’s beta AI Functions let SQL queries call an LLM or embedding provider directly, keeping classification, extraction, generation, translation, redaction, filtering, embedding, and similarity work alongside stored data. The feature addresses workflows that export rows to separate LLM, orchestration, or vector-database systems and then reload results, with the stated goal of running the full RAG cycle in ClickHouse. Functions make remote HTTP calls per input, or batch embedding inputs, and return native ClickHouse values; in open-source deployments, named collections store provider credentials and endpoint settings. Examples use Hacker News data for topic classification and natural-language filtering, while the guidance emphasizes cheap predicates, quotas, LIMIT tests, prompt-injection defenses, non-determinism, cost, latency, and provider access to data in cleartext after TLS termination.
Uber · · Problem & solutionUber Eats rebuilt its search pipeline after accumulated architectural decisions made latency increasingly costly across retrieval, hydration, ranking, ads, and presentation. The team changed its primary metric from backend API response time to Above-the-Fold completion, then introduced paginated responses with server-side caching and concurrent template rendering; these changes improved ATF by over 200 milliseconds. Other work removed low-yield retrieval strategies, grouped items by product identity to reduce lookups by over 100 times, split ranking and presentation hydration, eliminated false DAG dependencies, added request hedging, and redesigned ad data as column-oriented in memory. The combined effort cut search latency in half, including approximately 120 milliseconds from retrieval, 100-plus from hydration changes, and 130 from ads, while planned product-based search, zero-pass ranking, and streaming target further gains.
Uber · · Problem & solutionUber describes how it made Apache Pinot clusters resilient to zone failures while preserving query and ingestion availability for Tier-0 use cases. It maps Uber’s virtual resource isolation group IDs to Pinot server pools, uses pool-based instance assignment with replica-group segment assignment, and requires at least as many isolation groups as the table’s replication factor. During migration, automated pool registration, configuration backfills, granular APIs, selective rebalancing, and rollback support were used across more than 400 Tier 0-1 clusters and thousands of tables. An internal drill found that one remaining replica continued serving traffic without query degradation after a zone went down, while isolation-group-based claim and release policies enabled parallel restarts and sequential rollouts with baking time.
Grab · · Problem & solutionPartner Gateway needed low-latency analytics for API traffic and response-time metrics across intervals from one hour to 30-day retrospectives, with a particular dataset potentially reaching approximately 6.8 billion rows in a 30-day period. Apache Pinot was selected as a distributed real-time OLAP datastore, using high-throughput ingestion and columnar, indexed, partitioned data for millisecond analytical queries. Initial aggregation queries on datasets over 150GB timed out for 20- to 30-day periods, exceeding 10 seconds and threatening a 300-millisecond SLA. The implementation routes metrics through Kafka and Apache Flink, then ingests Schema Registry-compatible messages into Pinot, while partitioning by metric name and adding aggregation-focused optimizations. For one metric, query latency fell from 5,072 to 35 milliseconds; Star-tree indexing improved speed but raised 30-day storage from 192GB to 373GB, with the additional AWS EBS cost described as minor.
More like this
AI-DLC is an AI-driven development lifecycle that runs delivery through Inception, Design, and Construction, with specs, steering files, and hooks keeping agents aligned to agreed intent.
Apple Silicon's defining choice is one shared memory pool for CPU, GPU, and Neural Engine. This diagram maps that fabric, the on-package bandwidth it buys, and the scaling limits it imposes.
Antimicrobial resistance surveillance is a layered data architecture: raw reads and demand signals flow in, get normalized into governed tables, and answer plain-language queries. The diagram maps each boundary.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
